Package: visuallm
Conditional Language Model Generation Visualizationwhen evaluating language models it is often pain to see what is generated and whythis little package is avue.jsfrontend together withflaskbackend and it is designed to easily show some interesting visualizations on conditional generation modelsit handles frontend-backend communication as well as frontend renderinghence the developper can focus only on ML aspects of his work!VERSION:0.4.5reload all the elements on button pressroute changes handled by:keyattribute (to follow the actual best practices, improves scrolling experience)changelog:0.4.3added table modal, instead of creating a file that will be downloaded acopy to clipboardbutton was addedchangelog:0.4.0Text can be written with Bold and Italic Markdown (i.e. if you writeplain_text_element.content = "**text** ...", it will automatically be translated to<b>text</b> ...)Horizontal Multi Radio Element (multiple radios in horizontal row)if navbar overflows, it overflows to scrollon loading the collapsible expands which fixes bugTable of contentInstallationUsageExamplesAlpaca ExamplePersonaChat ExampleDocumentation LinksAcknowledgementInstallationinstall from pypi:pip install visuallmExample UsageAlpaca ExampleThe first workflow that we'll show is the workflow where you don't alter the implementation of the components at all and just use the provided components.Run InstructionsThe alpaca example code can be found here:./examples_py/alpaca_example, the code can be started by runningflask --app examples_py.alpaca_example.app run.We'll usealpacadataset andgpt2model as those are reasonably small to run even on less performant computers.# ./examples_py/alpaca_example/app.py lines 14-19defcreate_text_to_tokenizer(loaded_sample,target:str|None=None)->str:text_to_tokenizer=f"Instruction:{loaded_sample['instruction']}Answer:"iftargetisnotNone:text_to_tokenizer+=" "+targetAll the datasets are different, therefore we expect the user to provide 3 functions, which
define how the text which is tokenized is constructed, how the text for the one step prediction
is constructed, and how the target text is constructed.# ./examples_py/alpaca_example/app.py lines 22-41defcreate_text_to_tokenizer_one_step(loaded_sample,received_tokens:list[str])->str:# one step prediction means that the model is used to predict tokens one per one# received_tokens list contains already selected tokenstext_to_tokenizer=(f"Instruction:{loaded_sample['instruction']}Answer:"+"".join(received_tokens))returntext_to_tokenizerdefretrieve_target_str(loaded_sample):returnloaded_sample["output"]generator=HuggingFaceGenerator(model=model,tokenizer=tokenizer,create_text_to_tokenizer=create_text_to_tokenizer,Instantiate all the components from the library and run the server# ./examples_py/alpaca_example/app.py lines 44-57)app=create_app(dataset=dataset,generator_choices={"gpt2":generator})Dataset Visualization (Screenshots)In the screenshot, you can see that the dataset browser is created, where you can select dataset sample and dataset split and the frontend will show the inputs to the model and also the expected output.Generation (Screenshots)In the screenshot, you can see the dataset browser and the model generations.Next Token Prediction (Screenshots)In the screenshot, you can see that the library enables you to go through the generation step by step and explore why the generated sample (which can be seen e.g. on the Generation tab) looks the way it looks, how the distribution is skewed, etc.PersonaChat ExampleThe second workflow that we'll show is the workflow where you alter the implementation of the components, so that the dataset sample is shown in a different way.If you want to use the app with the personachat dataset, you can play with prepared example by running:flask --app examples_py.persona_chat_example.app run.The code for the sample can be found here:./examples_py/persona_chat_example.CustomizationThe personachat dataset contains two pieces of information for each dataset sample.The bot's personaThe past dialogue historySo we will add aTableElementwhich will display the two tables, one with bot's persona and one with past dialogue history. Since the visualization code is the same for all the components we will extract it into a separate class.# ./examples_py/persona_chat_example/components/input_display.py lines 9-55classPersonaChatVisualization:def__init__(self)->None:# just for the typechecker to not complainself.loaded_sample:Any=1definit_dialogue_vis_elements(self)->list[ElementBase]:"""Init elements which display the personachat tables."""table_input_heading=HeadingElement(content="Structure of Dialogue")self.input_table_vis=TableElement()return[table_input_heading,self.input_table_vis]defupdate_dialogue_structure_display(self,add_target:bool=True):"""Update elements which display the personachat tables."""sample=self.loaded_samplecontext=copy.deepcopy(sample["history"])ifadd_target:context.append(sample["candidates"][-1])persona=sample["personality"]self.set_sample_tables_element(persona,context)defset_sample_tables_element(self,persona:list[str],context:list[str],other_last:bool=False):"""Populate the tables with the information from the dataset sample."""self.input_table_vis.clear()self.input_table_vis.add_table(title="BOT Persona",headers=["Trait"],rows=[[t]fortinpersona],)d_len=len(context)# dialogue lengthbot_on_odd=int(d_len%2==(1ifnotother_lastelse0))whos=["BOT"ifi%2==bot_on_oddelse"OTHER"foriinrange(d_len)]iflen(context)>0:self.input_table_vis.add_table("Turns",["Who","Turn"],[[w,u]forw,uinzip(whos,context,strict=True)],)Afterwards we need to implement the inheritors of components that should make use of this specific visualization of the dataset sample. Here is an example of theGenerationcomponent.# ./examples_py/persona_chat_example/components/generation.py lines 1-23fromvisuallm.componentsimportGenerationComponentfromvisuallm.elements.element_baseimportElementBasefrom.input_displayimportPersonaChatVisualizationclassGeneration(GenerationComponent,PersonaChatVisualization):def__post_init__(self,*args,**kwargs):self.after_on_generator_change_callback()definit_model_input_display(self)->list[ElementBase]:return[*PersonaChatVisualization.init_dialogue_vis_elements(self),*super().init_model_input_display(),]defupdate_model_input_display(self):super().update_model_input_display()PersonaChatVisualization.update_dialogue_structure_display(self,add_target=False)Generation PlaygroundSelect which parameters you want to use for generation, plug in aHuggingFacemodel, or anOpenAItoken and have fun with experimenting with various generation hyperparameters!Chat PlaygroundSelect which parameters you want to use for generation, plug in aHuggingFacemodel, or anOpenAItoken and have fun with chatting with the model!Visualize Next Token PredictionsBy usingvisuallm.components.NextTokenPredictionComponent.NextTokenPredictionComponentyou can just plug the HuggingFace model in and go through the generation process step by step.Other ExamplesThere is some other documentation:How does the communication and bootstrapping of the components work ? (link)What is a minimal app that can be constructed ? (link)How do the elements work, how can I create custom components ? (link)Acknowledgementthis work was published on the INLG 2023 conference as a demo paper (link will be added later)Supported by the project TL05000236 AI asistent pro žáky a učitele co-financed by the Technological Agency of the Czech Republic and
by the ERC (No. 101039303 NG-NLG). Resources provided by the LINDAT/CLARIAH-CZ Research Infrastructure