Package: evo_researcher
Evo ResearcherOverviewThis project is aimed to be an iteration on top ofAutonolas AI Mechs'sresearch process
aimed towards making informed predictions.It contains two primary features:An information research functionAn information grading functionAdditionally, Mech's predict capability has been ported into a function that can also be independently run from this repo.Below, there's a high level explanation of their implementations, respectively.Research FunctionThe research function takes a question, like"Will Twitter implement a new misinformation policy before the 2024 elections?"and will then:Generate n web search queriesRe-rank the queries using an LLM call, and then select the most relevant onesSearch the web for each query, usingTavilyScrape and sanitize the content of each result's websiteUse Langchain'sRecursiveCharacterTextSplitterto split the content of all pages into chunks.Create embeddings of all chunks, and store the source of each as metadataIterate over the queries selected on step2. And for each one of them, vector search for the most relevant embeddings for each.Aggregate the chunks from the previous steps and prepare a report.Grading FunctionFor the implmentation of this function, the information quality criteria were selected fromhttps://guides.lib.unc.edu/evaluating-info/evaluate, ignoringusabilityandintended audience.Upon receiving a question like"Will Twitter implement a new misinformation policy before the 2024 elections?"and information, it will:Create en evaluation planPerform the evaluation of the information according to the plan from the previous stepExtract the scores from the evaluationPredictPorted implementation from:https://github.com/valory-xyz/mech/blob/main/tools/prediction_request_embedding/prediction_sentence_embedding.pyInstallationpoetryinstall
poetryshellRunResearchWith Evo:poetryrunpython./evo_researcher/main.pyresearch"Will Twitter implement a new misinformation policy before the 2024 elections?"evoWith Autonolas:poetryrunpython./evo_researcher/main.pyresearch"Will Twitter implement a new misinformation policy before the 2024 elections?"autonolasPredictpoetryrunpython./evo_researcher/main.pypredict"Will Twitter implement a new misinformation policy before the 2024 elections?"./outputs/myinfopathEvaluatepoetryrunpython./evo_researcher/main.pyevaluate"Will Twitter implement a new misinformation policy before the 2024 elections?"./outputs/myinfopathTestRun all questionspytestRun specific questionsUsepytest's-kflag and a string matcher. Example:pytest-k"Twitter"Example resultsEvo Reports:https://hackmd.io/mrCRBJyiTi-aO1gSFPjoDgEvo Information excerpts:https://hackmd.io/VQGJgHD1SImZrDR7puauJA?bothAutonolas Information excerpts:https://hackmd.io/5qt_0HkvQyuGZ2r0RqJtoQIdeas for future improvementFor the researcher:Using LLM re-ranking, likeCursorto optimize context-space and reduce noiseUseself-consistencyand generate several reports and compare them to choose the best, or even merge informationPlan research using more complex techniques liketree of thoughtsImplement a research loop, where research is performed and then evaluated. If the evaluation scores are under certain threshold, re-iterate to gather missing information or different sources, etc.Perform web searches under different topic or category focuses likeTavilydoes. For example, some questions benefit more from a "social media focused" research: gathering information from twitter threads, blog articles. Others benefit more from prioritizing scientific papers, institutional statements, and so on.Identify strong claims and perform sub-searches to verify them. This is the basis of AI powered fact-checkers like:https://fullfact.org/Evaluate sources credibilityFurther iterate over chunking and vector-search strategiesUseHyDEFor the information evaluator/graderUseself-consistencyto generate several scores and choose the most repeated ones.Enhance the evaluation and reduce its biases through the implementation of more advanced techniques, like the ones described herehttps://arxiv.org/pdf/2307.03025.pdfand herehttps://arxiv.org/pdf/2305.17926.pdfFurther evaluate biases towards writing-style, length, among others described here:https://arxiv.org/pdf/2308.02575.pdfand mitigate themEvaluate using different evaluation criteria