Package: decontext
QA DecontextualizationSeeexperimentsfor description of how to run experiments investigating this method.InstallationpipinstalldecontextOr if you prefer to install locally:condacreate-ndecontextpython=3.9
condaactivatedecontext
gitclonehttps://github.com/bnewm0609/qa-decontextualization.git
pipinstall-e.Quick StartSet your OpenAI API keyexportOPENAI_API_KEY='yourkey'By default, all requests to APIs are cached at$HOME/.cache/decontext, but this can be changed by setting the environment variableexport DECONTEXT_CACHE_DIR=path/to/cache.DecontextualizeTo decontextualize a snippet using some context, you can pass both the snippet and context to the decontextualization function.
Currently, the expected format for the context isentire full-text papers.
These include the title, abstract, and the sections of the paper.
The title, abstract, and full text are all required fields.
ThePaperContextclass, which holds these full-texts is a Pydantic model, so its values can be parsed fromjsonstrings as shown below.fromdecontextimportPaperContext,Sectioncontext=PaperContext(title="Identifying Dogmatism in Social Media: Signals and Models",abstract="We explore linguistic and behavioral features of dogmatism in social media and construct statistical models that can identify dogmatic comments. Our model is based on a corpus of Reddit posts, collected across a diverse set of conversational topics and annotated via paid crowdsourcing. We operationalize key aspects of dogmatism described by existing psychology theories (such as over-confidence), finding they have predictive power. We also find evidence for new signals of dogmatism, such as the tendency of dogmatic posts to refrain from signaling cognitive processes. When we use our predictive model to analyze millions of other Reddit posts, we find evidence that suggests dogmatism is a deeper personality trait, present for dogmatic users across many different domains, and that users who engage on dogmatic comments tend to show increases in dogmatic posts themselves.",full_text=[Section(section_name="Introduction",paragraphs=["<paragraph 1>","<paragraph 2>",...]),...],)snippet="Concretely, we apply the BOW+LING model trained on the full Reddit dataset to millions of new unannotated posts, labeling these posts with a probability of dogmatism according to the classifier (0=non-dogmatic, 1=dogmatic)."decontext(snippet=snippet,context=context)Subsection names should be separated from their supersection name with ":::". For example, the subsection "Metrics" of the "Methods" section would have thesection_name: "Methods ::: Metrics".The context can be loaded in using thePaperContext.parse_rawmethod:PaperContext.parse_raw("""{"title": "<title>","abstract": "<abstract>","full_text": [{"section_name" : "<section_title>","paragraphs": ["<paragraph>", ...]}, ...]}""")Additionally, thedecontextfunction also supports using multiple papers as context:decontext(snippet=snippet,context=paper_1_context,additional_context=[paper_2_context])The argumentcontextshould be the one that contains the snippet. The argumentadditional_contextcan contain other potentially useful material (e.g. papers that are cited in the snippet).DebuggingFor debugging purposes, it's useful to have access to the intermediate outputs of the pipeline. To show these, set thereturn_metadataargument toTrue. The returned metadata is an instance ofdecontext.PaperSnippet, which contains these outputs along with the cost of the run.new_snippet,metadata=decontext(snippet,paper_1,return_metadata=True)>PaperSnippet({"idx":"<unique identifier for the snippet>","snippet":"<original snippet>","context":"<context used in decontextualization>","question":[{"qid":"<question_1_id>","question":"<question_1>","answer":"<answer>","evidence":[{"section":"<section_name>","paragraph":"<paragraph>"},...]},...]"decontextualized_snippet":"<snippet after decontextualization>""cost":<cost_of_run_in_USD>})Customizing Pipeline ComponentsIf you want to use your own question generation, question answering, or synthesis models as part of the pipeline, you can incorporate them easily.
Each step of the pipeline takes adecontext.PaperSnippetinstance. This is the data structure that pipeline operates over. Each step fills in a field of thePaperSnippet.question generation fills inPaperSnippet.qae.questionby callingPaperSnippet.add_question(question)question answering optionally fills inPaperSnippet.qae.Evidencethrough retrieval by callingPaperSnippet.add_additional_paragraphs(), and fills inPaperSnippet.qae.answersby callingPaperSnippet.add_answer(). (qaestands for "Question, Answer, Evidence").synthesis fills inPaperSnippet.decontextualized_snippetby callingPaperSnippet.add_decontextualized_snippet()The custom component must call the relevant function for it's part.Your custom component should inherit from thedecontext.PipelineStepclass and override therunmethod. The method takes only one argument - thePaperSnippetobject. See thedecontex/steps/{qgen,qa,synth}.pyfor examples. this is fineUnder the hood, thedecontextmethod does the following:# 1. Creates the Pipeline objectpipeline=Pipeline(steps=[TemplateQGenStep(),TemplateQAStep(),TemplateSynthStep(),])# 2. Create the PaperSnippet objectps=PaperSnippet(snippet=snippet,context=context,qae=[])# 3. Runs each component of the pipeline in orderforstepinpipeline.steps:step.run(ps)Let's say you define your own Question Generation pipeline using a template that's better suited for your data than the default.fromdecontext.modelimportload_modelfromdecontext.templateimportTemplatefromdecontext.step.stepimportQGenStepclassCustomQGen(QGenStep):def__init__(self):self.gpt3=load_model("text-davinci-003")self.template=Template("""\Ask clarifying questions about the sinppet that comes from this:Title: {{title}}Abstract: {{abstract}}Snippet: {{snippet}}Questions:-""")defrun(paper_snippet:PaperSnippet):prompt=self.template.fill({"title":paper_snippet.context.title,"abstract":paper_snippet.context.abstract,"snippet":paper_snippet.snippet,})response=self.gpt3(prompt)forquestioninresponse:paper_snippet.add_question(question=response[0])Then, you can incorporate it into the pipeline and pass the pipeline to thedecontextfunction:fromdecontextimportPipelinepipeline=Pipeline(steps=[CustomQGen(),TemplateQAStep(),TemplateSynthStep(),])decontext(snippet=snippet,context=context,pipeline=pipeline)The templates used to prompt OpenAI models for the default Pipeline areyamlfiles defined insrc/decontext/templates.CachingBy default, all requests are cached to save credit. Responses are cached globally, regardless of user and across all requests in the file specified by$DECONTEXT_CACHE_DIRusing thediskcachelibrary. However, there are situations where you don't want to cache such as when trying to test out different parts. There are four different levels of caching, that are defined indecontext.cache.CacheStateNO_CACHE- No caching is done at allINVALIDATE- If there's a cached response, it's ignored and the API is hit again. The new response is cached.NORMAL- If there's a cached response, it's returned. Otherwise the API is hit and the response is cached.ENFORCE_CACHE- If there's no cached response, an exception is raised.There are three ways to control the caching:At thepipelinelevel:  e.g.decontext(snippet, context, cache_state=CacheState.ENFORCE_CACHE)sets the cache state for all queries.A step can be initialized with acache_stateparameter to set the default caching behavior for the step. E.g.,TemplateQGenStep(cache_state=CacheState.ENFORCE_CACHE)A step can be run with acache_stateto override its default caching behavior. E.g.TemplateQGenStep().run(snippet, cache_state=CacheState.ENFORCE_CACHE).The default cache state isCacheState.NORMAL.Note that currently, the cost returned as part of the metadata ignores caching - the value is how much the prompt would have cost if the response was not cached.Function Declarationdefdecontext(snippet:str,context:PaperContext,additional_contexts:Optional[List[PaperContext]]=None,pipeline:Optional[Pipeline]=None,return_metadata:bool=False,cache_states:Optional[Union[CacheState,List[Optional[CacheState]]]]=None,)->Union[str,Tuple[str,PaperSnippet]]:"""Decontextualizes the snippet using the given context according to the given config.Args:snippet: The text snippet to decontextualize.context: The context to incorporate into the decontextualization. This context must include the snippet.additional_contexts: Additional context to use in the decontextualization (eg papers that are cited in the snippet).pipeline: The pipeline to run on the snippet. If not provided, a default retrieval-based pipeline will be run.return_metadata: Flag for returning the PaperSnippet object with intermediate outputs. (See below).cache_states: The cache states to use for each step of the pipeline. If None, the default cache stateis used. If a single CacheState is given, it is used for all steps. If a list of CacheStates is given,the ith CacheState is used for the ith step.Returns:string with the decontextualized version of the snippet.if `return_metadata = True`, additionally return the intermediate results for each step of the pipelineas described above."""