Package: repr-llm
repr-llmCreate lightweight representations of Python objects for Large Language Model consumptionBackgroundIn Python, we have a way to represent our objects within interpreters:repr.In IPython, it goes even further. We can register rich represenations of plots, tables, and all kinds of objects. As an example, developers can augment their objects with a_repr_html_method to expose a rich HTML version of their object. The most common example most Pythonistas know about is showing tables for their data inside notebooks viapandas.importpandasaspddf=pd.DataFrame({"a":[1,2,3],"b":[4,5,6]})dfab014125236This is a great way to show data in a notebook for humans. What if there was a way to provide a compact yet rich representation for Large Language Models?The IdeaTherepr_llmpackage introduces the idea that python objects can emit a rich representation for LLM consumption. With the advent ofOpenAI's Code Interpreter,Noteable plugin for ChatGPT, andLangChain's Python REPL Tool, we have massive opportunity to create rich visualizations for humans and rich text for models.Let's begin by creating aBookclass that has a regular__repr__and a_repr_llm_:classBook:def__init__(self,title,author,year,genre):self.title=titleself.author=authorself.year=yearself.genre=genredef__repr__(self):returnf"Book('{self.title}', '{self.author}',{self.year}, '{self.genre}')"def_repr_llm_(self):return(f"A Book object representing '{self.title}' by{self.author}, "f"published in the year{self.year}. Genre:{self.genre}. "f"Instantiated with `{repr(self)}`")fromrepr_llmimportregister_llm_formatterip=get_ipython()# Current IPython shellregister_llm_formatter(ip)# This is how IPython creates the Out[*] prompt in the notebookdata,_=ip.display_formatter.format(Book('Attack of the Black Rectangles','Amy Sarig King',2022,"Middle Grade"))data{"text/plain":"Book('Attack of the Black Rectangles', 'Amy Sarig King', 2022, 'Middle Grade')","text/llm+plain":"A Book object representing 'Attack of the Black Rectangles' by Amy Sarig King, published in the year 2022. Genre: Middle Grade. Instantiated with `Book('Attack of the Black Rectangles', 'Amy Sarig King', 2022, 'Middle Grade')`"}How it worksTherepr_llmpackage provides aregister_llm_formatterfunction that takes an IPython shell and registers a new formatter for thetext/llm+plainmimetype.When IPython goes to display an object, it will first check if the object has a_repr_llm_method. If it does, it will call that method and include the result as part of the representation for the object.FAQWhy not just use_repr_markdown_? (or__repr__)The_repr_markdown_method is a great way to show rich text in a notebook. The reason is that it's meant for humans to read. Large Language Models can read that too. However, there are going to be times whenMarkdownis too big for the model (token limit) or too complex (too many tokens to understand).Originally I was going to suggest more package authors use_repr_markdown_(and they should!). ThenPeter Wangsuggested that we have a version written for the models, just like OpenAI'sai-plugin.jsondoes, especially since the LLMscanbe a more advanced reader.Any LLM user can still use_repr_markdown_to show rich text to the model. This provides an extra option that isexplicit. It's a way for developers to say "this is what I want the model to see".Where does thetext/llm+plainmimetype come from?The mimetype is a convention being proposed. It's a way to say "this is a plain text representation for a Large Language Model". It's not a standard (yet!). It's a way to say "this is what I want the model to see" while we explore this space.Are there examples of this in the wild?ChatGPT plugins provide something very similar to repr vs repr_llm with theapi_plugin_jsonand OpenAPI specs, especially since there's a delineation ofdescription_for_humananddescription_for_model.How did this originate?While experimenting with Large Language Models directly💬🤗and with tools likegenai,dangermode, andlangchainI've been naturally converting representations of my data or text to markdown as a format that GPT models can understand and converse with a user about.What's next?Convince as many library authors as possible, just like we did with_repr_html_to use_repr_llm_to provide a lightweight representation of their objects that is:Deterministic - no side effectsNavigable - states what other functions can be run to get more information, create better plots, etc.Lightweight - take into account how much text a GPT model can readSafe - no secrets, no PII, no sensitive dataCreditThank you toDave Shoupfor the conversations about pandas representation and how we can keep improving what we send forOut[*]to Large Language Models.