Package: takeoff-haystack
Integration: TitanMLUse TitanML's Takeoff server to serve local models efficiently with Haystack 2.0IntroductionYou can use the Takeoff inference server to deploy local models efficiently in your Haystack 2.0 pipelines. Takeoff is a state-of-the art inference server focused on deploying openly available language models at scale. It can run LLMs on local machines with consumer GPUs, and on cloud infrastructure.The TakeoffGenerator component in Haystack 2.0 is a wrapper around the Takeoff server API, and can be used to serve takeoff-deployed models efficiently in Haystack pipelines.Installationpipinstalltakeoff_haystackUsageYou can interact with takeoff deployed models using theTakeoffGeneratorcomponent in Haystack. To do so, you must have a takeoff model deployed. For information on how to do so, please read the takeoff docshere.The following example deploys a gpt2 model using takeoff locally on port 3000.dockerrun--gpusall-eTAKEOFF_MODEL_NAME=TheBloke/Llama-2-7B-Chat-AWQ\-eTAKEOFF_DEVICE=cuda\-eTAKEOFF_MAX_SEQUENCE_LENGTH=256\-it\-p3000:3000tytn/takeoff-pro:0.11.0-gpuTextGenerationBelow is an example of using takeoff models in a Haystack RAG pipeline. It summarizes headlines from popular news sites in technology.fromtypingimportDict,List
fromhaystackimportDocument,Pipeline
fromhaystack.components.builders.prompt_builderimportPromptBuilderfromhaystack.components.retrievers.in_memoryimportInMemoryBM25Retriever
fromhaystack.document_stores.in_memoryimportInMemoryDocumentStore
importfeedparser
fromtakeoff_haystackimportTakeoffGenerator# Dict of website RSS feedsurls={'theverge':'https://www.theverge.com/rss/frontpage/','techcrunch':'https://techcrunch.com/feed','mashable':'https://mashable.com/feeds/rss/all','cnet':'https://cnet.com/rss/news','engadget':'https://engadget.com/rss.xml','zdnet':'https://zdnet.com/news/rss.xml','venturebeat':'https://feeds.feedburner.com/venturebeat/SZYF','readwrite':'https://readwrite.com/feed/','wired':'https://wired.com/feed/rss','gizmodo':'https://gizmodo.com/rss',}# Configurable parametersNUM_WEBSITES=3NUM_TITLES=1defget_titles(urls:Dict[str,str],num_sites:int,num_titles:int)->List[str]:titles:List[str]=[]sites=list(urls.keys())[:num_sites]forsiteinsites:feed=feedparser.parse(urls[site])entries=feed.entries[:num_titles]forentryinentries:titles.append(entry.title)returntitlestitles=get_titles(urls,NUM_WEBSITES,NUM_TITLES)titles_string=" - ".join(titles)document_store=InMemoryDocumentStore()document_store.write_documents([Document(content=titles_string)])template="""HEADLINES:{% for document in documents %}{{ document.content }}{% endfor %}REQUEST: {{ query }}"""pipe=Pipeline()pipe.add_component("retriever",InMemoryBM25Retriever(document_store=document_store))pipe.add_component("prompt_builder",PromptBuilder(template=template))pipe.add_component("llm",TakeoffGenerator(base_url="http://localhost",port="3000"))pipe.connect("retriever","prompt_builder.documents")pipe.connect("prompt_builder","llm")query=f"Summarize each of the {NUM_WEBSITES * NUM_TITLES} provided headlines in three words."response=pipe.run({"prompt_builder":{"query":query},"retriever":{"query":query}})print(response["llm"]["replies"])You should see a response like the following['\n\n\nANSWER:\n\n1. Poker Roguelike - Exciting gameplay\n2. AI-powered news reader - Personalized feed\n3. Best laptops MWC 2024 - Powerful devices']