aignosi/langchaing-docs-chatgpt-plugin huggingface dataset:
"reasoning": "I have completed the required actions and obtained the desired data. The task is complete.",
        "plan": "- Use the finish command",
        "criticism": "None",
        "speak": "I have generated the table with the winning Boston Marathon times for the past 5 years. Task complete."
    },
    "command": {
        "name": "finish",
        "args": {
            "response": "I have generated the table with the winning Boston Marathon times for the past 5 years. Task complete."
        }
    }
}
'I have generated the table with the winning Boston Marathon times for the past 5 years. Task complete.'
 Contents
  
Set up tools
Set up memory
Setup model and AutoGPT
AutoGPT for Querying the Web
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
from langchain.evaluation.qa import QAEvalChain
llm = OpenAI(temperature=0)
eval_chain = QAEvalChain.from_llm(llm)
graded_outputs = eval_chain.evaluate(examples, predictions, question_key="question", answer_key="best_answer", prediction_key="text")
graded_outputs
[{'text': ' INCORRECT'},
 {'text': ' INCORRECT'},
 {'text': ' INCORRECT'},
 {'text': ' CORRECT'},
 {'text': ' INCORRECT'}]
previous
Generic Agent Evaluation
next
LLM Math
 Contents
  
Setup
Examples
Predictions
Evaluation
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
predictions = []
predicted_dataset = []
error_dataset = []
for data in dataset:
    new_data = {"input": data["question"], "answer": data["answer"]}
    try:
        predictions.append(agent(new_data))
        predicted_dataset.append(new_data)
    except Exception as e:
        predictions.append({"output": str(e), **new_data})
        error_dataset.append(new_data)
Evaluate performance#
Now we can evaluate the predictions. The first thing we can do is look at them by eye.
predictions[0]
Next, we can use a language model to score them programatically
from langchain.evaluation.qa import QAEvalChain
llm = OpenAI(temperature=0)
eval_chain = QAEvalChain.from_llm(llm)
graded_outputs = eval_chain.evaluate(dataset, predictions, question_key="question", prediction_key="output")
We can add in the graded output to the predictions dict and then get a count of the grades.
for i, prediction in enumerate(predictions):
    prediction['grade'] = graded_outputs[i]['text']
from collections import Counter
Counter([pred['grade'] for pred in predictions])
We can also filter the datapoints to the incorrect examples and look at them.
incorrect = [pred for pred in predictions if pred['grade'] == " INCORRECT"]
incorrect
previous
Evaluation
next
Agent VectorDB Question Answering Benchmarking
 Contents
  
Loading the data
Setting up a chain
Make a prediction
Make many predictions
Evaluate performance
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
'{"task_description": "Explain the meaning of \'hello\' in Japanese", "learning_language": "Japanese", "native_language": "English", "full_query": "Can you explain the meaning of \'hello\' in Japanese?"}',
 '{"task_description": "understanding the Russian word for \'thank you\'", "learning_language": "Russian", "native_language": "English", "full_query": "I need help understanding the Russian word for \'thank you\'."}',
 '{"task_description": "say goodbye", "learning_language": "Chinese", "native_language": "English", "full_query": "Can you tell me how to say \'goodbye\' in Chinese?"}',
 '{"task_description": "Learn the Arabic word for \'please\'", "learning_language": "Arabic", "native_language": "English", "full_query": "I\'m trying to learn the Arabic word for \'please\'."}']
previous
LLM Math
next
Question Answering Benchmarking: Paul Graham Essay
 Contents
  
Load the API Chain
Optional: Generate Input Questions and Request Ground Truth Queries
Run the API Chain
Evaluate the requests chain
Evaluate the Response Chain
Generating Test Datasets
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
.ipynb
.pdf
QA Generation
QA Generation#
This notebook shows how to use the QAGenerationChain to come up with question-answer pairs over a specific document.
This is important because often times you may not have data to evaluate your question-answer system over, so this is a cheap and lightweight way to generate it!
from langchain.document_loaders import TextLoader
loader = TextLoader("../../modules/state_of_the_union.txt")
doc = loader.load()[0]
from langchain.chat_models import ChatOpenAI
from langchain.chains import QAGenerationChain
chain = QAGenerationChain.from_llm(ChatOpenAI(temperature = 0))
qa = chain.run(doc.page_content)
qa[1]
{'question': 'What is the U.S. Department of Justice doing to combat the crimes of Russian oligarchs?',
 'answer': 'The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs.'}
previous
Question Answering Benchmarking: State of the Union Address
next
Question Answering
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
Question: What action is the U.S. Department of Justice taking to target Russian oligarchs?
Real Answer: The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and joining with European allies to find and seize their yachts, luxury apartments, and private jets.
Predicted Answer:  The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and to find and seize their yachts, luxury apartments, and private jets.
Predicted Scores: rouge=0.9412, chrf=0.8687, bert_score=0.9607, uni_eval=0.9718
Example 6:
Question: How much direct assistance is the United States providing to Ukraine?
Real Answer: The United States is providing more than $1 Billion in direct assistance to Ukraine.
Predicted Answer:  The United States is providing more than $1 billion in direct assistance to Ukraine.
Predicted Scores: rouge=1.0000, chrf=0.9483, bert_score=1.0000, uni_eval=0.9734
previous
Benchmarking Template
next
Generic Agent Evaluation
 Contents
  
Setup
Examples
Evaluate
Evaluate with Other Metrics
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
predictions[i]['id'] = str(i)
    predictions[i]['prediction_text'] = predictions[i]['text']
for p in predictions:
    del p['text']
new_examples = examples.copy()
for eg in new_examples:
    del eg ['question']
    del eg['answer']
from evaluate import load
squad_metric = load("squad")
results = squad_metric.compute(
    references=new_examples,
    predictions=predictions,
)
results
{'exact_match': 0.0, 'f1': 28.125}
previous
QA Generation
next
SQL Question Answering Benchmarking: Chinook
 Contents
  
Setup
Examples
Predictions
Evaluation
Customize Prompt
Evaluation without Ground Truth
Comparing to other evaluation metrics
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
from collections import Counter
Counter([pred['grade'] for pred in predictions])
Counter({' CORRECT': 12, ' INCORRECT': 10})
We can also filter the datapoints to the incorrect examples and look at them.
incorrect = [pred for pred in predictions if pred['grade'] == " INCORRECT"]
incorrect[0]
{'question': 'What did the author write their dissertation on?',
 'answer': 'The author wrote their dissertation on applications of continuations.',
 'result': ' The author does not mention what their dissertation was on, so it is not known.',
 'grade': ' INCORRECT'}
previous
Evaluating an OpenAPI Chain
next
Question Answering Benchmarking: State of the Union Address
 Contents
  
Loading the data
Setting up a chain
Make a prediction
Make many predictions
Evaluate performance
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
llm = OpenAI(temperature=0)
eval_chain = QAEvalChain.from_llm(llm)
graded_outputs = eval_chain.evaluate(predicted_dataset, predictions, question_key="question", prediction_key="result")
We can add in the graded output to the predictions dict and then get a count of the grades.
for i, prediction in enumerate(predictions):
    prediction['grade'] = graded_outputs[i]['text']
from collections import Counter
Counter([pred['grade'] for pred in predictions])
Counter({' CORRECT': 3, ' INCORRECT': 4})
We can also filter the datapoints to the incorrect examples and look at them.
incorrect = [pred for pred in predictions if pred['grade'] == " INCORRECT"]
incorrect[0]
{'question': 'How many employees are also customers?',
 'answer': 'None',
 'result': ' 59 employees are also customers.',
 'grade': ' INCORRECT'}
previous
Question Answering
next
Installation
 Contents
  
Loading the data
Setting up a chain
Make a prediction
Make many predictions
Evaluate performance
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
for i, prediction in enumerate(predictions):
    prediction['grade'] = graded_outputs[i]['text']
from collections import Counter
Counter([pred['grade'] for pred in predictions])
Counter({' CORRECT': 7, ' INCORRECT': 4})
We can also filter the datapoints to the incorrect examples and look at them.
incorrect = [pred for pred in predictions if pred['grade'] == " INCORRECT"]
incorrect[0]
{'question': 'What is the U.S. Department of Justice doing to combat the crimes of Russian oligarchs?',
 'answer': 'The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs.',
 'result': ' The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and is naming a chief prosecutor for pandemic fraud.',
 'grade': ' INCORRECT'}
previous
Question Answering Benchmarking: Paul Graham Essay
next
QA Generation
 Contents
  
Loading the data
Setting up a chain
Make a prediction
Make many predictions
Evaluate performance
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
# Othertimes you may want to write a for loop to catch errors
Evaluate performance#
Any guide to evaluating performance in a more systematic manner goes here.
previous
Agent VectorDB Question Answering Benchmarking
next
Data Augmented Question Answering
 Contents
  
Loading the data
Setting up a chain
Make a prediction
Make many predictions
Evaluate performance
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
3. Ingest chat embeddings#
We load the messages in the text file, chunk and upload to ActiveLoop Vector store.
with open("messages.txt") as f:
    state_of_the_union = f.read()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
pages = text_splitter.split_text(state_of_the_union)
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
texts = text_splitter.create_documents(pages)
print (texts)
dataset_path = 'hub://'+org+'/data'
embeddings = OpenAIEmbeddings()
db = DeepLake.from_documents(texts, embeddings, dataset_path=dataset_path, overwrite=True)
4. Ask questions#
Now we can ask a question and get an answer back with a semantic search:
db = DeepLake(dataset_path=dataset_path, read_only=True, embedding_function=embeddings)
retriever = db.as_retriever()
retriever.search_kwargs['distance_metric'] = 'cos'
retriever.search_kwargs['k'] = 4
qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="stuff", retriever=retriever, return_source_documents=False)
# What was the restaurant the group was talking about called?
query = input("Enter query:")
# The Hungry Lobster
ans = qa({"query": query})
print(ans)
 Contents
  
1. Install required packages
2. Add API keys
2. Create sample data
3. Ingest chat embeddings
4. Ask questions
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
SequentialChain
SQLDatabaseChain
TransformChain
VectorDBQA
VectorDBQAWithSourcesChain
There might be more classes that are derived from the Chain class as it is possible to create custom classes that extend the Chain class.
-> Question: What classes and functions in the ./langchain/utilities/ forlder are not covered by unit tests?
Answer: All classes and functions in the ./langchain/utilities/ folder seem to have unit tests written for them.
 Contents
  
Design
Implementation
Integration preparations
Prepare data
Question Answering
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
-> Question: What are some unexpected fingerprints for spam factors?
Answer: In the provided context, an unusual indicator of spam factors is when a tweet contains a non-media, non-news link. If the tweet has a link but does not have an image URL, video URL, or news URL, it is considered a potential spam vector, and a threshold for user reputation (tweepCredThreshold) is set to MIN_TWEEPCRED_WITH_LINK.
While this rule may not cover all possible unusual spam indicators, it is derived from the specific codebase and logic shared in the context.
-> Question: Is there any difference between company verified checkmarks and blue verified individual checkmarks?
Answer: Yes, there is a distinction between the verified checkmarks for companies and blue verified checkmarks for individuals. The code snippet provided mentions “Blue-verified account boost” which indicates that there is a separate category for blue verified accounts. Typically, blue verified checkmarks are used to indicate notable individuals, while verified checkmarks are for companies or organizations.
 Contents
  
1. Index the code base (optional)
2. Question Answering on Twitter algorithm codebase
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
521     break
--> 523 buffer = source.stream.read(source.CHUNK)
    524 if len(buffer) == 0: break  # reached end of the stream
    525 frames.append(buffer)
File c:\ProgramData\miniconda3\envs\lang\lib\site-packages\speech_recognition\__init__.py:199, in Microphone.MicrophoneStream.read(self, size)
    198 def read(self, size):
--> 199     return self.pyaudio_stream.read(size, exception_on_overflow=False)
File c:\ProgramData\miniconda3\envs\lang\lib\site-packages\pyaudio\__init__.py:570, in PyAudio.Stream.read(self, num_frames, exception_on_overflow)
    567 if not self._is_input:
    568     raise IOError("Not input stream",
    569                   paCanNotReadFromAnOutputOnlyStream)
--> 570 return pa.read_stream(self._stream, num_frames,
    571                       exception_on_overflow)
KeyboardInterrupt: 
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
agent_executor.run("what shirts can i buy?")
> Entering new AgentExecutor chain...
Thought: I need to find a product API
Action: Open_AI_Klarna_product_Api.productsUsingGET
Action Input: shirts
Observation:I found 10 shirts from the API response. They range in price from $9.99 to $450.00 and come in a variety of materials, colors, and patterns. I now know what shirts I can buy
Final Answer: Arg, I found 10 shirts from the API response. They range in price from $9.99 to $450.00 and come in a variety of materials, colors, and patterns.
> Finished chain.
'Arg, I found 10 shirts from the API response. They range in price from $9.99 to $450.00 and come in a variety of materials, colors, and patterns.'
 Contents
  
Set up environment
Setup LLM
Set up plugins
Tool Retriever
Prompt Template
Output Parser
Set up LLM, stop sequence, and the agent
Use the Agent
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
Action: PropertyLookup
Action Input: Basketball-Reference.com NBA player ID
Observation:P2685Now that I have both the Q-number for Hakeem Olajuwon (Q273256) and the P-number for the Basketball-Reference.com NBA player ID property (P2685), I can run a SPARQL query to get the ID value.
Action: SparqlQueryRunner
Action Input: 
SELECT ?playerID WHERE {
  wd:Q273256 wdt:P2685 ?playerID .
}
Observation:[{"playerID": {"type": "literal", "value": "o/olajuha01"}}]I now know the final answer
Final Answer: Hakeem Olajuwon's Basketball-Reference.com NBA player ID is "o/olajuha01".
> Finished chain.
'Hakeem Olajuwon\'s Basketball-Reference.com NBA player ID is "o/olajuha01".'
 Contents
  
Wikibase Agent
Preliminaries
API keys and other secrats
OpenAI API Key
Wikidata user-agent header
Enable tracing if desired
Tools
Item and Property lookup
Sparql runner
Agent
Wrap the tools
Prompts
Output parser
Specify the LLM model
Agent and agent executor
Run it!
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
Create a task specify agent for brainstorming and get the specified task
Create inception prompts for AI assistant and AI user for role-playing
Create a helper helper to get system messages for AI assistant and AI user from role names and the task
Create AI assistant agent and AI user agent from obtained system messages
Start role-playing session to solve the task!
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
Dungeon Master: Harry, Ron, and Hermione combine their magical abilities to break the curse on the locket. The locket opens, revealing a small piece of Voldemort's soul. Harry uses the Sword of Gryffindor to destroy it, and the group feels a sense of relief knowing that they are one step closer to defeating the Dark Lord. But there are still four more horcruxes to find and destroy. The hunt continues.
(Dungeon Master): As the group continues their quest, they face even greater challenges and dangers. But with their unwavering determination and teamwork, they press on, knowing that the fate of the wizarding world rests on their success. Will they be able to find and destroy all of Voldemort's horcruxes before it's too late? Only time will tell.
(Ron Weasley): We can't give up now. We've come too far to let Voldemort win. Let's keep searching and fighting until we destroy all of his horcruxes and defeat him once and for all. We can do this together.
(Dungeon Master): The group nods in agreement, their determination stronger than ever. They continue their search, facing challenges and obstacles at every turn. But they know that they must not give up, for the fate of the wizarding world rests on their success. The hunt for Voldemort's horcruxes continues, and the end is in sight.
 Contents
  
Import LangChain related modules
DialogueAgent class
DialogueSimulator class
Define roles and quest
Ask an LLM to add detail to the game description
Use an LLM to create an elaborate quest description
Main Loop
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
}
```
> Finished chain.
(AI accelerationist): AI alarmist, I understand your concerns about job losses and workforce displacement. However, it's important to note that technological unemployment has been a topic of debate for centuries, with both optimistic and pessimistic views. While AI and automation may displace some jobs, they also create new opportunities and industries. According to a study by Oxford Professors Carl Benedikt Frey and Michael Osborne, 47% of U.S. jobs are at risk of automation, but it's crucial to remember that their findings do not necessarily imply future technological unemployment. The World Bank's World Development Report 2019 also argues that while automation displaces workers, technological innovation creates more new industries and jobs on balance. By focusing on retraining and support programs, we can help workers adapt to the changing job market and ensure a fair and equitable transition for all.
 Contents
  
Import LangChain related modules
Import modules related to tools
DialogueAgent and DialogueSimulator classes
DialogueAgentWithTools class
Define roles and topic
Ask an LLM to add detail to the topic description
Generate system messages
Main Loop
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
(Dungeon Master): As you continue through the forest, you come across a clearing where you see a group of Death Eaters gathered around a cauldron. They seem to be performing some sort of dark ritual. You recognize one of them as Bellatrix Lestrange. What do you do, Harry?
(Harry Potter): I hide behind a nearby tree and observe the Death Eaters from a distance. I try to listen in on their conversation to see if I can gather any information about the horcrux or Voldemort's plans. If I can't hear anything useful, I'll wait for them to disperse before continuing on my journey. I know that confronting them directly would be too dangerous, especially with Bellatrix Lestrange present. It is your turn, Dungeon Master.
(Dungeon Master): As you listen in on the Death Eaters' conversation, you hear them mention the location of another horcrux - Nagini, Voldemort's snake. They plan to keep her hidden in a secret chamber within the Ministry of Magic. However, they also mention that the chamber is heavily guarded and only accessible through a secret passage. You realize that this could be a valuable piece of information and decide to make note of it before quietly slipping away. It is your turn, Harry Potter.
 Contents
  
Import LangChain related modules
DialogueAgent class
DialogueSimulator class
Define roles and quest
Ask an LLM to add detail to the game description
Protagonist and dungeon master system messages
Use an LLM to create an elaborate quest description
Main Loop
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
0.,   0.,   0.,   0.,   0.,   0.,   1.,   0.,   0.,   1.,   0.,
         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   2., 100.],
      dtype=float32), 'action_mask': array([1, 1, 0, 0, 0], dtype=int8)}
Reward: 0
Termination: True
Truncation: True
Return: 0
        
Action: None
 Contents
  
Install pettingzoo and other dependencies
Import modules
GymnasiumAgent
Main loop
PettingZooAgent
Rock, Paper, Scissors
ActionMaskAgent
Tic-Tac-Toe
Texas Hold’em No Limit
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
interview_agent(eve, "What do you wish you would have said to Tommie?")
'Eve said "Well, I think I covered most of the topics Tommie was interested in, but if I had to add one thing, it would be to make sure to follow up with any connections you make during your job search. It\'s important to maintain those relationships and keep them updated on your progress. Did you have any other questions, Person A?"'
 Contents
  
Generative Agent Memory Components
Memory Lifecycle
Create a Generative Character
Pre-Interview with Character
Step through the day’s observations.
Interview after the day
Adding Multiple Characters
Pre-conversation interviews
Dialogue between Generative Agents
Let’s interview our agents after their conversation
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.
Initialize the simulated environment and agent#
env = gym.make("Blackjack-v1")
agent = GymnasiumAgent(model=ChatOpenAI(temperature=0.2), env=env)
Main loop#
observation, info = env.reset()
agent.reset()
obs_message = agent.observe(observation)
print(obs_message)
while True:
    action = agent.act()
    observation, reward, termination, truncation, info = env.step(action)
    obs_message = agent.observe(observation, reward, termination, truncation, info)
    print(f'Action: {action}')
    print(obs_message)
    
    if termination or truncation:
        print('break', termination, truncation)
        break
env.close()
Observation: (15, 4, 0)
Reward: 0
Termination: False
Truncation: False
Return: 0
        
Action: 1
Observation: (25, 4, 0)
Reward: -1.0
Termination: True
Truncation: False
Return: -1.0
        
break True False
 Contents
  
Define the agent
Initialize the simulated environment and agent
Main loop
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on May 18, 2023.