{'action': 'on_llm_start', 'name': 'OpenAI', 'step': 3, 'starts': 2, 'ends': 1, 'errors': 0, 'text_ctr': 0, 'chain_starts': 0, 'chain_ends': 0, 'llm_starts': 2, 'llm_ends': 1, 'llm_streams': 0, 'tool_starts': 0, 'tool_ends': 0, 'agent_ends': 0, 'prompts': 'Tell me a joke'}
{'action': 'on_llm_start', 'name': 'OpenAI', 'step': 3, 'starts': 2, 'ends': 1, 'errors': 0, 'text_ctr': 0, 'chain_starts': 0, 'chain_ends': 0, 'llm_starts': 2, 'llm_ends': 1, 'llm_streams': 0, 'tool_starts': 0, 'tool_ends': 0, 'agent_ends': 0, 'prompts': 'Tell me a poem'}


{'action': 'on_llm_end', 'token_usage_prompt_tokens': 24, 'token_usage_completion_tokens': 138, 'token_usage_total_tokens': 162, 'model_name': 'text-davinci-003', 'step': 4, 'starts': 2, 'ends': 2, 'errors': 0, 'text_ctr': 0, 'chain_starts': 0, 'chain_ends': 0, 'llm_starts': 2, 'llm_ends': 2, 'llm_streams': 0, 'tool_starts': 0, 'tool_ends': 0, 'agent_ends': 0, 'text': '\n\nQ: What did the fish say when it hit the wall?\nA: Dam!', 'generation_info_finish_reason': 'stop', 'generation_info_logprobs': None, 'flesch_reading_ease': 109.04, 'flesch_kincaid_grade': 1.3, 'smog_index': 0.0, 'coleman_liau_index': -1.24, 'automated_readability_index': 0.3, 'dale_chall_readability_score': 5.5, 'difficult_words': 0, 'linsear_write_formula': 5.5, 'gunning_fog': 5.2, 'text_standard': '5th and 6th grade', 'fernandez_huerta': 133.58, 'szigriszt_pazos': 131.54, 'gutierrez_polini': 62.3, 'crawford': -0.2, 'gulpease_index': 79.8, 'osman': 116.91}


{'action': 'on_llm_end', 'token_usage_prompt_tokens': 24, 'token_usage_completion_tokens': 138, 'token_usage_total_tokens': 162, 'model_name': 'text-davinci-003', 'step': 4, 'starts': 2, 'ends': 2, 'errors': 0, 'text_ctr': 0, 'chain_starts': 0, 'chain_ends': 0, 'llm_starts': 2, 'llm_ends': 2, 'llm_streams': 0, 'tool_starts': 0, 'tool_ends': 0, 'agent_ends': 0, 'text': '\n\nRoses are red,\nViolets are blue,\nSugar is sweet,\nAnd so are you.', 'generation_info_finish_reason': 'stop', 'generation_info_logprobs': None, 'flesch_reading_ease': 83.66, 'flesch_kincaid_grade': 4.8, 'smog_index': 0.0, 'coleman_liau_index': 3.23, 'automated_readability_index': 3.9, 'dale_chall_readability_score': 6.71, 'difficult_words': 2, 'linsear_write_formula': 6.5, 'gunning_fog': 8.28, 'text_standard': '6th and 7th grade', 'fernandez_huerta': 115.58, 'szigriszt_pazos': 112.37, 'gutierrez_polini': 54.83, 'crawford': 1.4, 'gulpease_index': 72.1, 'osman': 100.17}


{'action': 'on_llm_end', 'token_usage_prompt_tokens': 24, 'token_usage_completion_tokens': 138, 'token_usage_total_tokens': 162, 'model_name': 'text-davinci-003', 'step': 4, 'starts': 2, 'ends': 2, 'errors': 0, 'text_ctr': 0, 'chain_starts': 0, 'chain_ends': 0, 'llm_starts': 2, 'llm_ends': 2, 'llm_streams': 0, 'tool_starts': 0, 'tool_ends': 0, 'agent_ends': 0, 'text': '\n\nRoses are red,\nViolets are blue,\nSugar is sweet,\nAnd so are you.', 'generation_info_finish_reason': 'stop', 'generation_info_logprobs': None, 'flesch_reading_ease': 83.66, 'flesch_kincaid_grade': 4.8, 'smog_index': 0.0, 'coleman_liau_index': 3.23, 'automated_readability_index': 3.9, 'dale_chall_readability_score': 6.71, 'difficult_words': 2, 'linsear_write_formula': 6.5, 'gunning_fog': 8.28, 'text_standard': '6th and 7th grade', 'fernandez_huerta': 115.58, 'szigriszt_pazos': 112.37, 'gutierrez_polini': 54.83, 'crawford': 1.4, 'gulpease_index': 72.1, 'osman': 100.17}
{'action_records':           action    name  step  starts  ends  errors  text_ctr  chain_starts  \


0   on_llm_start  OpenAI     1       1     0       0         0             0
1   on_llm_start  OpenAI     1       1     0       0         0             0
2   on_llm_start  OpenAI     1       1     0       0         0             0
3   on_llm_start  OpenAI     1       1     0       0         0             0
4   on_llm_start  OpenAI     1       1     0       0         0             0
5   on_llm_start  OpenAI     1       1     0       0         0             0
6     on_llm_end     NaN     2       1     1       0         0             0
7     on_llm_end     NaN     2       1     1       0         0             0
8     on_llm_end     NaN     2       1     1       0         0             0
9     on_llm_end     NaN     2       1     1       0         0             0
10    on_llm_end     NaN     2       1     1       0         0             0
11    on_llm_end     NaN     2       1     1       0         0             0
12  on_llm_start  OpenAI     3       2     1       0         0             0
13  on_llm_start  OpenAI     3       2     1       0         0             0


14  on_llm_start  OpenAI     3       2     1       0         0             0
15  on_llm_start  OpenAI     3       2     1       0         0             0
16  on_llm_start  OpenAI     3       2     1       0         0             0
17  on_llm_start  OpenAI     3       2     1       0         0             0
18    on_llm_end     NaN     4       2     2       0         0             0
19    on_llm_end     NaN     4       2     2       0         0             0
20    on_llm_end     NaN     4       2     2       0         0             0
21    on_llm_end     NaN     4       2     2       0         0             0
22    on_llm_end     NaN     4       2     2       0         0             0
23    on_llm_end     NaN     4       2     2       0         0             0
    chain_ends  llm_starts  ...  difficult_words  linsear_write_formula  \
0            0           1  ...              NaN                    NaN
1            0           1  ...              NaN                    NaN
2            0           1  ...              NaN                    NaN
3            0           1  ...              NaN                    NaN
4            0           1  ...              NaN                    NaN
5            0           1  ...              NaN                    NaN


5            0           1  ...              NaN                    NaN
6            0           1  ...              0.0                    5.5
7            0           1  ...              2.0                    6.5
8            0           1  ...              0.0                    5.5
9            0           1  ...              2.0                    6.5
10           0           1  ...              0.0                    5.5
11           0           1  ...              2.0                    6.5
12           0           2  ...              NaN                    NaN
13           0           2  ...              NaN                    NaN
14           0           2  ...              NaN                    NaN
15           0           2  ...              NaN                    NaN
16           0           2  ...              NaN                    NaN
17           0           2  ...              NaN                    NaN
18           0           2  ...              0.0                    5.5
19           0           2  ...              2.0                    6.5
20           0           2  ...              0.0                    5.5
21           0           2  ...              2.0                    6.5
22           0           2  ...              0.0                    5.5
23           0           2  ...              2.0                    6.5
    gunning_fog      text_standard  fernandez_huerta szigriszt_pazos  \
0           NaN                NaN               NaN             NaN


0           NaN                NaN               NaN             NaN
1           NaN                NaN               NaN             NaN
2           NaN                NaN               NaN             NaN
3           NaN                NaN               NaN             NaN
4           NaN                NaN               NaN             NaN
5           NaN                NaN               NaN             NaN
6          5.20  5th and 6th grade            133.58          131.54
7          8.28  6th and 7th grade            115.58          112.37
8          5.20  5th and 6th grade            133.58          131.54
9          8.28  6th and 7th grade            115.58          112.37
10         5.20  5th and 6th grade            133.58          131.54
11         8.28  6th and 7th grade            115.58          112.37
12          NaN                NaN               NaN             NaN
13          NaN                NaN               NaN             NaN
14          NaN                NaN               NaN             NaN
15          NaN                NaN               NaN             NaN
16          NaN                NaN               NaN             NaN
17          NaN                NaN               NaN             NaN
18         5.20  5th and 6th grade            133.58          131.54
19         8.28  6th and 7th grade            115.58          112.37
20         5.20  5th and 6th grade            133.58          131.54


21         8.28  6th and 7th grade            115.58          112.37
22         5.20  5th and 6th grade            133.58          131.54
23         8.28  6th and 7th grade            115.58          112.37
    gutierrez_polini  crawford  gulpease_index   osman
0                NaN       NaN             NaN     NaN
1                NaN       NaN             NaN     NaN
2                NaN       NaN             NaN     NaN
3                NaN       NaN             NaN     NaN
4                NaN       NaN             NaN     NaN
5                NaN       NaN             NaN     NaN
6              62.30      -0.2            79.8  116.91
7              54.83       1.4            72.1  100.17
8              62.30      -0.2            79.8  116.91
9              54.83       1.4            72.1  100.17
10             62.30      -0.2            79.8  116.91
11             54.83       1.4            72.1  100.17
12               NaN       NaN             NaN     NaN
13               NaN       NaN             NaN     NaN
14               NaN       NaN             NaN     NaN
15               NaN       NaN             NaN     NaN
16               NaN       NaN             NaN     NaN
17               NaN       NaN             NaN     NaN
18             62.30      -0.2            79.8  116.91


19             54.83       1.4            72.1  100.17
20             62.30      -0.2            79.8  116.91
21             54.83       1.4            72.1  100.17
22             62.30      -0.2            79.8  116.91
23             54.83       1.4            72.1  100.17
[24 rows x 39 columns], 'session_analysis':     prompt_step         prompts    name  output_step  \
0             1  Tell me a joke  OpenAI            2
1             1  Tell me a poem  OpenAI            2
2             1  Tell me a joke  OpenAI            2
3             1  Tell me a poem  OpenAI            2
4             1  Tell me a joke  OpenAI            2
5             1  Tell me a poem  OpenAI            2
6             3  Tell me a joke  OpenAI            4
7             3  Tell me a poem  OpenAI            4
8             3  Tell me a joke  OpenAI            4
9             3  Tell me a poem  OpenAI            4
10            3  Tell me a joke  OpenAI            4
11            3  Tell me a poem  OpenAI            4
                                               output  \
0   \n\nQ: What did the fish say when it hit the w...
1   \n\nRoses are red,\nViolets are blue,\nSugar i...


2   \n\nQ: What did the fish say when it hit the w...
3   \n\nRoses are red,\nViolets are blue,\nSugar i...
4   \n\nQ: What did the fish say when it hit the w...
5   \n\nRoses are red,\nViolets are blue,\nSugar i...
6   \n\nQ: What did the fish say when it hit the w...
7   \n\nRoses are red,\nViolets are blue,\nSugar i...
8   \n\nQ: What did the fish say when it hit the w...
9   \n\nRoses are red,\nViolets are blue,\nSugar i...
10  \n\nQ: What did the fish say when it hit the w...
11  \n\nRoses are red,\nViolets are blue,\nSugar i...
    token_usage_total_tokens  token_usage_prompt_tokens  \
0                        162                         24
1                        162                         24
2                        162                         24
3                        162                         24
4                        162                         24
5                        162                         24
6                        162                         24
7                        162                         24
8                        162                         24
9                        162                         24
10                       162                         24
11                       162                         24
    token_usage_completion_tokens  flesch_reading_ease  flesch_kincaid_grade  \
0                             138               109.04                   1.3
1                             138                83.66                   4.8


1                             138                83.66                   4.8
2                             138               109.04                   1.3
3                             138                83.66                   4.8
4                             138               109.04                   1.3
5                             138                83.66                   4.8
6                             138               109.04                   1.3
7                             138                83.66                   4.8
8                             138               109.04                   1.3
9                             138                83.66                   4.8
10                            138               109.04                   1.3
11                            138                83.66                   4.8
    ...  difficult_words  linsear_write_formula  gunning_fog  \
0   ...                0                    5.5         5.20
1   ...                2                    6.5         8.28
2   ...                0                    5.5         5.20
3   ...                2                    6.5         8.28
4   ...                0                    5.5         5.20
5   ...                2                    6.5         8.28
6   ...                0                    5.5         5.20
7   ...                2                    6.5         8.28
8   ...                0                    5.5         5.20
9   ...                2                    6.5         8.28
10  ...                0                    5.5         5.20


10  ...                0                    5.5         5.20
11  ...                2                    6.5         8.28
        text_standard  fernandez_huerta  szigriszt_pazos  gutierrez_polini  \
0   5th and 6th grade            133.58           131.54             62.30
1   6th and 7th grade            115.58           112.37             54.83
2   5th and 6th grade            133.58           131.54             62.30
3   6th and 7th grade            115.58           112.37             54.83
4   5th and 6th grade            133.58           131.54             62.30
5   6th and 7th grade            115.58           112.37             54.83
6   5th and 6th grade            133.58           131.54             62.30
7   6th and 7th grade            115.58           112.37             54.83
8   5th and 6th grade            133.58           131.54             62.30
9   6th and 7th grade            115.58           112.37             54.83
10  5th and 6th grade            133.58           131.54             62.30
11  6th and 7th grade            115.58           112.37             54.83
   crawford  gulpease_index   osman


crawford  gulpease_index   osman
0      -0.2            79.8  116.91
1       1.4            72.1  100.17
2      -0.2            79.8  116.91
3       1.4            72.1  100.17
4      -0.2            79.8  116.91
5       1.4            72.1  100.17
6      -0.2            79.8  116.91
7       1.4            72.1  100.17
8      -0.2            79.8  116.91
9       1.4            72.1  100.17
10     -0.2            79.8  116.91
11      1.4            72.1  100.17
[12 rows x 24 columns]}
2023-03-29 14:00:25,948 - clearml.Task - INFO - Completed model upload to https://files.clear.ml/langchain_callback_demo/llm.988bd727b0e94a29a3ac0ee526813545/models/simple_sequential
At this point you can already go to https://app.clear.ml and take a look at the resulting ClearML Task that was created.
Among others, you should see that this notebook is saved along with any git information. The model JSON that contains the used parameters is saved as an artifact, there are also console logs and under the plots section, you’ll find tables that represent the flow of the chain.
Finally, if you enabled visualizations, these are stored as HTML files under debug samples.


Finally, if you enabled visualizations, these are stored as HTML files under debug samples.
Scenario 2: Creating an agent with tools#
To show a more advanced workflow, let’s create an agent with access to tools. The way ClearML tracks the results is not different though, only the table will look slightly different as there are other types of actions taken when compared to the earlier, simpler example.
You can now also see the use of the finish=True keyword, which will fully close the ClearML Task, instead of just resetting the parameters and prompts for a new conversation.
from langchain.agents import initialize_agent, load_tools
from langchain.agents import AgentType
# SCENARIO 2 - Agent with Tools
tools = load_tools(["serpapi", "llm-math"], llm=llm, callbacks=callbacks)
agent = initialize_agent(
    tools,
    llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    callbacks=callbacks,
)
agent.run(
    "Who is the wife of the person who sang summer of 69?"
)
clearml_callback.flush_tracker(langchain_asset=agent, name="Agent with Tools", finish=True)
> Entering new AgentExecutor chain...
{'action': 'on_chain_start', 'name': 'AgentExecutor', 'step': 1, 'starts': 1, 'ends': 0, 'errors': 0, 'text_ctr': 0, 'chain_starts': 1, 'chain_ends': 0, 'llm_starts': 0, 'llm_ends': 0, 'llm_streams': 0, 'tool_starts': 0, 'tool_ends': 0, 'agent_ends': 0, 'input': 'Who is the wife of the person who sang summer of 69?'}


{'action': 'on_llm_start', 'name': 'OpenAI', 'step': 2, 'starts': 2, 'ends': 0, 'errors': 0, 'text_ctr': 0, 'chain_starts': 1, 'chain_ends': 0, 'llm_starts': 1, 'llm_ends': 0, 'llm_streams': 0, 'tool_starts': 0, 'tool_ends': 0, 'agent_ends': 0, 'prompts': 'Answer the following questions as best you can. You have access to the following tools:\n\nSearch: A search engine. Useful for when you need to answer questions about current events. Input should be a search query.\nCalculator: Useful for when you need to answer questions about math.\n\nUse the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [Search, Calculator]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can repeat N times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin!\n\nQuestion: Who is the wife of the person who sang summer of 69?\nThought:'}


{'action': 'on_llm_end', 'token_usage_prompt_tokens': 189, 'token_usage_completion_tokens': 34, 'token_usage_total_tokens': 223, 'model_name': 'text-davinci-003', 'step': 3, 'starts': 2, 'ends': 1, 'errors': 0, 'text_ctr': 0, 'chain_starts': 1, 'chain_ends': 0, 'llm_starts': 1, 'llm_ends': 1, 'llm_streams': 0, 'tool_starts': 0, 'tool_ends': 0, 'agent_ends': 0, 'text': ' I need to find out who sang summer of 69 and then find out who their wife is.\nAction: Search\nAction Input: "Who sang summer of 69"', 'generation_info_finish_reason': 'stop', 'generation_info_logprobs': None, 'flesch_reading_ease': 91.61, 'flesch_kincaid_grade': 3.8, 'smog_index': 0.0, 'coleman_liau_index': 3.41, 'automated_readability_index': 3.5, 'dale_chall_readability_score': 6.06, 'difficult_words': 2, 'linsear_write_formula': 5.75, 'gunning_fog': 5.4, 'text_standard': '3rd and 4th grade', 'fernandez_huerta': 121.07, 'szigriszt_pazos': 119.5, 'gutierrez_polini': 54.91, 'crawford': 0.9, 'gulpease_index': 72.7, 'osman': 92.16}


I need to find out who sang summer of 69 and then find out who their wife is.
Action: Search
Action Input: "Who sang summer of 69"{'action': 'on_agent_action', 'tool': 'Search', 'tool_input': 'Who sang summer of 69', 'log': ' I need to find out who sang summer of 69 and then find out who their wife is.\nAction: Search\nAction Input: "Who sang summer of 69"', 'step': 4, 'starts': 3, 'ends': 1, 'errors': 0, 'text_ctr': 0, 'chain_starts': 1, 'chain_ends': 0, 'llm_starts': 1, 'llm_ends': 1, 'llm_streams': 0, 'tool_starts': 1, 'tool_ends': 0, 'agent_ends': 0}
{'action': 'on_tool_start', 'input_str': 'Who sang summer of 69', 'name': 'Search', 'description': 'A search engine. Useful for when you need to answer questions about current events. Input should be a search query.', 'step': 5, 'starts': 4, 'ends': 1, 'errors': 0, 'text_ctr': 0, 'chain_starts': 1, 'chain_ends': 0, 'llm_starts': 1, 'llm_ends': 1, 'llm_streams': 0, 'tool_starts': 2, 'tool_ends': 0, 'agent_ends': 0}
Observation: Bryan Adams - Summer Of 69 (Official Music Video).


Observation: Bryan Adams - Summer Of 69 (Official Music Video).
Thought:{'action': 'on_tool_end', 'output': 'Bryan Adams - Summer Of 69 (Official Music Video).', 'step': 6, 'starts': 4, 'ends': 2, 'errors': 0, 'text_ctr': 0, 'chain_starts': 1, 'chain_ends': 0, 'llm_starts': 1, 'llm_ends': 1, 'llm_streams': 0, 'tool_starts': 2, 'tool_ends': 1, 'agent_ends': 0}


{'action': 'on_llm_start', 'name': 'OpenAI', 'step': 7, 'starts': 5, 'ends': 2, 'errors': 0, 'text_ctr': 0, 'chain_starts': 1, 'chain_ends': 0, 'llm_starts': 2, 'llm_ends': 1, 'llm_streams': 0, 'tool_starts': 2, 'tool_ends': 1, 'agent_ends': 0, 'prompts': 'Answer the following questions as best you can. You have access to the following tools:\n\nSearch: A search engine. Useful for when you need to answer questions about current events. Input should be a search query.\nCalculator: Useful for when you need to answer questions about math.\n\nUse the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [Search, Calculator]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can repeat N times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin!\n\nQuestion: Who is the wife of the person who sang summer of 69?\nThought: I need to find out who sang summer of 69 and then find out who their wife is.\nAction: Search\nAction Input: "Who sang summer of 69"\nObservation: Bryan Adams - Summer Of 69 (Official Music Video).\nThought:'}


{'action': 'on_llm_end', 'token_usage_prompt_tokens': 242, 'token_usage_completion_tokens': 28, 'token_usage_total_tokens': 270, 'model_name': 'text-davinci-003', 'step': 8, 'starts': 5, 'ends': 3, 'errors': 0, 'text_ctr': 0, 'chain_starts': 1, 'chain_ends': 0, 'llm_starts': 2, 'llm_ends': 2, 'llm_streams': 0, 'tool_starts': 2, 'tool_ends': 1, 'agent_ends': 0, 'text': ' I need to find out who Bryan Adams is married to.\nAction: Search\nAction Input: "Who is Bryan Adams married to"', 'generation_info_finish_reason': 'stop', 'generation_info_logprobs': None, 'flesch_reading_ease': 94.66, 'flesch_kincaid_grade': 2.7, 'smog_index': 0.0, 'coleman_liau_index': 4.73, 'automated_readability_index': 4.0, 'dale_chall_readability_score': 7.16, 'difficult_words': 2, 'linsear_write_formula': 4.25, 'gunning_fog': 4.2, 'text_standard': '4th and 5th grade', 'fernandez_huerta': 124.13, 'szigriszt_pazos': 119.2, 'gutierrez_polini': 52.26, 'crawford': 0.7, 'gulpease_index': 74.7, 'osman': 84.2}
 I need to find out who Bryan Adams is married to.
Action: Search


I need to find out who Bryan Adams is married to.
Action: Search
Action Input: "Who is Bryan Adams married to"{'action': 'on_agent_action', 'tool': 'Search', 'tool_input': 'Who is Bryan Adams married to', 'log': ' I need to find out who Bryan Adams is married to.\nAction: Search\nAction Input: "Who is Bryan Adams married to"', 'step': 9, 'starts': 6, 'ends': 3, 'errors': 0, 'text_ctr': 0, 'chain_starts': 1, 'chain_ends': 0, 'llm_starts': 2, 'llm_ends': 2, 'llm_streams': 0, 'tool_starts': 3, 'tool_ends': 1, 'agent_ends': 0}
{'action': 'on_tool_start', 'input_str': 'Who is Bryan Adams married to', 'name': 'Search', 'description': 'A search engine. Useful for when you need to answer questions about current events. Input should be a search query.', 'step': 10, 'starts': 7, 'ends': 3, 'errors': 0, 'text_ctr': 0, 'chain_starts': 1, 'chain_ends': 0, 'llm_starts': 2, 'llm_ends': 2, 'llm_streams': 0, 'tool_starts': 4, 'tool_ends': 1, 'agent_ends': 0}
Observation: Bryan Adams has never married. In the 1990s, he was in a relationship with Danish model Cecilie Thomsen. In 2011, Bryan and Alicia Grimaldi, his ...


Thought:{'action': 'on_tool_end', 'output': 'Bryan Adams has never married. In the 1990s, he was in a relationship with Danish model Cecilie Thomsen. In 2011, Bryan and Alicia Grimaldi, his ...', 'step': 11, 'starts': 7, 'ends': 4, 'errors': 0, 'text_ctr': 0, 'chain_starts': 1, 'chain_ends': 0, 'llm_starts': 2, 'llm_ends': 2, 'llm_streams': 0, 'tool_starts': 4, 'tool_ends': 2, 'agent_ends': 0}


{'action': 'on_llm_start', 'name': 'OpenAI', 'step': 12, 'starts': 8, 'ends': 4, 'errors': 0, 'text_ctr': 0, 'chain_starts': 1, 'chain_ends': 0, 'llm_starts': 3, 'llm_ends': 2, 'llm_streams': 0, 'tool_starts': 4, 'tool_ends': 2, 'agent_ends': 0, 'prompts': 'Answer the following questions as best you can. You have access to the following tools:\n\nSearch: A search engine. Useful for when you need to answer questions about current events. Input should be a search query.\nCalculator: Useful for when you need to answer questions about math.\n\nUse the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [Search, Calculator]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can repeat N times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin!\n\nQuestion: Who


to the original input question\n\nBegin!\n\nQuestion: Who is the wife of the person who sang summer of 69?\nThought: I need to find out who sang summer of 69 and then find out who their wife is.\nAction: Search\nAction Input: "Who sang summer of 69"\nObservation: Bryan Adams - Summer Of 69 (Official Music Video).\nThought: I need to find out who Bryan Adams is married to.\nAction: Search\nAction Input: "Who is Bryan Adams married to"\nObservation: Bryan Adams has never married. In the 1990s, he was in a relationship with Danish model Cecilie Thomsen. In 2011, Bryan and Alicia Grimaldi, his ...\nThought:'}


{'action': 'on_llm_end', 'token_usage_prompt_tokens': 314, 'token_usage_completion_tokens': 18, 'token_usage_total_tokens': 332, 'model_name': 'text-davinci-003', 'step': 13, 'starts': 8, 'ends': 5, 'errors': 0, 'text_ctr': 0, 'chain_starts': 1, 'chain_ends': 0, 'llm_starts': 3, 'llm_ends': 3, 'llm_streams': 0, 'tool_starts': 4, 'tool_ends': 2, 'agent_ends': 0, 'text': ' I now know the final answer.\nFinal Answer: Bryan Adams has never been married.', 'generation_info_finish_reason': 'stop', 'generation_info_logprobs': None, 'flesch_reading_ease': 81.29, 'flesch_kincaid_grade': 3.7, 'smog_index': 0.0, 'coleman_liau_index': 5.75, 'automated_readability_index': 3.9, 'dale_chall_readability_score': 7.37, 'difficult_words': 1, 'linsear_write_formula': 2.5, 'gunning_fog': 2.8, 'text_standard': '3rd and 4th grade', 'fernandez_huerta': 115.7, 'szigriszt_pazos': 110.84, 'gutierrez_polini': 49.79, 'crawford': 0.7, 'gulpease_index': 85.4, 'osman': 83.14}
 I now know the final answer.
Final Answer: Bryan Adams has never been married.


I now know the final answer.
Final Answer: Bryan Adams has never been married.
{'action': 'on_agent_finish', 'output': 'Bryan Adams has never been married.', 'log': ' I now know the final answer.\nFinal Answer: Bryan Adams has never been married.', 'step': 14, 'starts': 8, 'ends': 6, 'errors': 0, 'text_ctr': 0, 'chain_starts': 1, 'chain_ends': 0, 'llm_starts': 3, 'llm_ends': 3, 'llm_streams': 0, 'tool_starts': 4, 'tool_ends': 2, 'agent_ends': 1}
> Finished chain.
{'action': 'on_chain_end', 'outputs': 'Bryan Adams has never been married.', 'step': 15, 'starts': 8, 'ends': 7, 'errors': 0, 'text_ctr': 0, 'chain_starts': 1, 'chain_ends': 1, 'llm_starts': 3, 'llm_ends': 3, 'llm_streams': 0, 'tool_starts': 4, 'tool_ends': 2, 'agent_ends': 1}
{'action_records':              action    name  step  starts  ends  errors  text_ctr  \
0      on_llm_start  OpenAI     1       1     0       0         0
1      on_llm_start  OpenAI     1       1     0       0         0
2      on_llm_start  OpenAI     1       1     0       0         0
3      on_llm_start  OpenAI     1       1     0       0         0


4      on_llm_start  OpenAI     1       1     0       0         0
..              ...     ...   ...     ...   ...     ...       ...
66      on_tool_end     NaN    11       7     4       0         0
67     on_llm_start  OpenAI    12       8     4       0         0
68       on_llm_end     NaN    13       8     5       0         0
69  on_agent_finish     NaN    14       8     6       0         0
70     on_chain_end     NaN    15       8     7       0         0
    chain_starts  chain_ends  llm_starts  ...  gulpease_index  osman  input  \
0              0           0           1  ...             NaN    NaN    NaN
1              0           0           1  ...             NaN    NaN    NaN
2              0           0           1  ...             NaN    NaN    NaN
3              0           0           1  ...             NaN    NaN    NaN
4              0           0           1  ...             NaN    NaN    NaN
..           ...         ...         ...  ...             ...    ...    ...
66             1           0           2  ...             NaN    NaN    NaN
67             1           0           3  ...             NaN    NaN    NaN
68             1           0           3  ...            85.4  83.14    NaN
69             1           0           3  ...             NaN    NaN    NaN


69             1           0           3  ...             NaN    NaN    NaN
70             1           1           3  ...             NaN    NaN    NaN
    tool  tool_input                                                log  \
0    NaN         NaN                                                NaN
1    NaN         NaN                                                NaN
2    NaN         NaN                                                NaN
3    NaN         NaN                                                NaN
4    NaN         NaN                                                NaN
..   ...         ...                                                ...
66   NaN         NaN                                                NaN
67   NaN         NaN                                                NaN
68   NaN         NaN                                                NaN
69   NaN         NaN   I now know the final answer.\nFinal Answer: B...
70   NaN         NaN                                                NaN
    input_str  description                                             output  \
0         NaN          NaN                                                NaN
1         NaN          NaN                                                NaN
2         NaN          NaN                                                NaN
3         NaN          NaN                                                NaN
4         NaN          NaN                                                NaN
..        ...          ...                                                ...
66        NaN          NaN  Bryan Adams has never married. In the 1990s, h...
67        NaN          NaN                                                NaN
68        NaN          NaN                                                NaN
69        NaN          NaN                Bryan Adams has never been married.
70        NaN          NaN                                                NaN
                                outputs
0                                   NaN
1                                   NaN
2                                   NaN
3                                   NaN
4                                   NaN
..                                  ...
66                                  NaN
67                                  NaN
68                                  NaN
69                                  NaN
70  Bryan Adams has never been married.
[71 rows x 47 columns], 'session_analysis':    prompt_step                                            prompts    name  \
0            2  Answer the following questions as best you can...  OpenAI


0            2  Answer the following questions as best you can...  OpenAI
1            7  Answer the following questions as best you can...  OpenAI
2           12  Answer the following questions as best you can...  OpenAI
   output_step                                             output  \
0            3   I need to find out who sang summer of 69 and ...
1            8   I need to find out who Bryan Adams is married...
2           13   I now know the final answer.\nFinal Answer: B...
   token_usage_total_tokens  token_usage_prompt_tokens  \
0                       223                        189
1                       270                        242
2                       332                        314
   token_usage_completion_tokens  flesch_reading_ease  flesch_kincaid_grade  \
0                             34                91.61                   3.8
1                             28                94.66                   2.7
2                             18                81.29                   3.7
   ...  difficult_words  linsear_write_formula  gunning_fog  \
0  ...                2                   5.75          5.4
1  ...                2                   4.25          4.2
2  ...                1                   2.50          2.8
       text_standard  fernandez_huerta  szigriszt_pazos  gutierrez_polini  \
0  3rd and 4th grade            121.07           119.50             54.91
1  4th and 5th grade            124.13           119.20             52.26


2  3rd and 4th grade            115.70           110.84             49.79
  crawford  gulpease_index  osman
0      0.9            72.7  92.16
1      0.7            74.7  84.20
2      0.7            85.4  83.14
[3 rows x 24 columns]}
Could not update last created model in Task 988bd727b0e94a29a3ac0ee526813545, Task status 'completed' cannot be updated
Tips and Next Steps#
Make sure you always use a unique name argument for the clearml_callback.flush_tracker function. If not, the model parameters used for a run will override the previous run!
If you close the ClearML Callback using clearml_callback.flush_tracker(..., finish=True) the Callback cannot be used anymore. Make a new one if you want to keep logging.
Check out the rest of the open source ClearML ecosystem, there is a data version manager, a remote execution agent, automated pipelines and much more!
previous
Chroma
next
Cohere
 Contents

Getting API Credentials
Setting Up
Scenario 1: Just an LLM
Scenario 2: Creating an agent with tools
Tips and Next Steps
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.md
.pdf
Metal
 Contents
What is Metal?
Quick start
Metal#
This page covers how to use Metal within LangChain.
What is Metal?#
Metal is a  managed retrieval & memory platform built for production. Easily index your data into Metal and run semantic search and retrieval on it.
Quick start#
Get started by creating a Metal account.
Then, you can easily take advantage of the MetalRetriever class to start retrieving your data for semantic search, prompting context, etc. This class takes a Metal instance and a dictionary of parameters to pass to the Metal API.
from langchain.retrievers import MetalRetriever
from metal_sdk.metal import Metal
metal = Metal("API_KEY", "CLIENT_ID", "INDEX_ID");
retriever = MetalRetriever(metal, params={"limit": 2})
docs = retriever.get_relevant_documents("search term")
previous
Llama.cpp
next
Milvus
 Contents

What is Metal?
Quick start
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.md
.pdf
RWKV-4
 Contents
Installation and Setup
Usage
RWKV
Model File
Rwkv-4 models -> recommended VRAM
RWKV-4#
This page covers how to use the RWKV-4 wrapper within LangChain.
It is broken into two parts: installation and setup, and then usage with an example.
Installation and Setup#
Install the Python package with pip install rwkv
Install the tokenizer Python package with pip install tokenizer
Download a RWKV model and place it in your desired directory
Download the tokens file
Usage#
RWKV#
To use the RWKV wrapper, you need to provide the path to the pre-trained model file and the tokenizer’s configuration.
from langchain.llms import RWKV
# Test the model
```python
def generate_prompt(instruction, input=None):
    if input:
        return f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
# Instruction:
{instruction}
# Input:
{input}
# Response:
"""
    else:
        return f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.
# Instruction:
{instruction}
# Response:
"""
model = RWKV(model="./models/RWKV-4-Raven-3B-v7-Eng-20230404-ctx4096.pth", strategy="cpu fp32", tokens_path="./rwkv/20B_tokenizer.json")
response = model(generate_prompt("Once upon a time, "))
Model File#
You can find links to model file downloads at the RWKV-4-Raven repository.
Rwkv-4 models -> recommended VRAM#
RWKV VRAM
Model | 8bit | bf16/fp16 | fp32


RWKV VRAM
Model | 8bit | bf16/fp16 | fp32
14B   | 16GB | 28GB      | >50GB
7B    | 8GB  | 14GB      | 28GB
3B    | 2.8GB| 6GB       | 12GB
1b5   | 1.3GB| 3GB       | 6GB
See the rwkv pip page for more information about strategies, including streaming and cuda support.
previous
Runhouse
next
SearxNG Search API
 Contents

Installation and Setup
Usage
RWKV
Model File
Rwkv-4 models -> recommended VRAM
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.md
.pdf
Replicate
 Contents
Installation and Setup
Calling a model
Replicate#
This page covers how to run models on Replicate within LangChain.
Installation and Setup#
Create a Replicate account. Get your API key and set it as an environment variable (REPLICATE_API_TOKEN)
Install the Replicate python client with pip install replicate
Calling a model#
Find a model on the Replicate explore page, and then paste in the model name and version in this format: owner-name/model-name:version
For example, for this dolly model, click on the API tab. The model name/version would be: "replicate/dolly-v2-12b:ef0e1aefc61f8e096ebe4db6b2bacc297daf2ef6899f0f7e001ec445893500e5"
Only the model param is required, but any other model parameters can also be passed in with the format input={model_param: value, ...}
For example, if we were running stable diffusion and wanted to change the image dimensions:
Replicate(model="stability-ai/stable-diffusion:db21e45d3f7023abc2a46ee38a23973f6dce16bb082a930b0c49861f96d1e5bf", input={'image_dimensions': '512x512'})
Note that only the first output of a model will be returned.
From here, we can initialize our model:
llm = Replicate(model="replicate/dolly-v2-12b:ef0e1aefc61f8e096ebe4db6b2bacc297daf2ef6899f0f7e001ec445893500e5")
And run it:
prompt = """


And run it:
prompt = """
Answer the following yes/no question by reasoning step by step.
Can a dog drive a car?
"""
llm(prompt)
We can call any Replicate model (not just LLMs) using this syntax. For example, we can call Stable Diffusion:
text2image = Replicate(model="stability-ai/stable-diffusion:db21e45d3f7023abc2a46ee38a23973f6dce16bb082a930b0c49861f96d1e5bf", input={'image_dimensions':'512x512'})
image_output = text2image("A cat riding a motorcycle by Picasso")
previous
Redis
next
Runhouse
 Contents

Installation and Setup
Calling a model
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.ipynb
.pdf
Aim
Aim#
Aim makes it super easy to visualize and debug LangChain executions. Aim tracks inputs and outputs of LLMs and tools, as well as actions of agents.
With Aim, you can easily debug and examine an individual execution:
Additionally, you have the option to compare multiple executions side by side:
Aim is fully open source, learn more about Aim on GitHub.
Let’s move forward and see how to enable and configure Aim callback.
Tracking LangChain Executions with AimIn this notebook we will explore three usage scenarios. To start off, we will install the necessary packages and import certain modules. Subsequently, we will configure two environment variables that can be established either within the Python script or through the terminal.
!pip install aim
!pip install langchain
!pip install openai
!pip install google-search-results
import os
from datetime import datetime
from langchain.llms import OpenAI
from langchain.callbacks import AimCallbackHandler, StdOutCallbackHandler
Our examples use a GPT model as the LLM, and OpenAI offers an API for this purpose. You can obtain the key from the following link: https://platform.openai.com/account/api-keys .
We will use the SerpApi to retrieve search results from Google. To acquire the SerpApi key, please go to https://serpapi.com/manage-api-key .
os.environ["OPENAI_API_KEY"] = "..."
os.environ["SERPAPI_API_KEY"] = "..."
The event methods of AimCallbackHandler accept the LangChain module or agent as input and log at least the prompts and generated results, as well as the serialized version of the LangChain module, to the designated Aim run.
session_group = datetime.now().strftime("%m.%d.%Y_%H.%M.%S")
aim_callback = AimCallbackHandler(
    repo=".",


aim_callback = AimCallbackHandler(
    repo=".",
    experiment_name="scenario 1: OpenAI LLM",
)
callbacks = [StdOutCallbackHandler(), aim_callback]
llm = OpenAI(temperature=0, callbacks=callbacks)
The flush_tracker function is used to record LangChain assets on Aim. By default, the session is reset rather than being terminated outright.
Scenario 1 In the first scenario, we will use OpenAI LLM.
# scenario 1 - LLM
llm_result = llm.generate(["Tell me a joke", "Tell me a poem"] * 3)
aim_callback.flush_tracker(
    langchain_asset=llm,
    experiment_name="scenario 2: Chain with multiple SubChains on multiple generations",
)
Scenario 2 Scenario two involves chaining with multiple SubChains across multiple generations.
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain
# scenario 2 - Chain
template = """You are a playwright. Given the title of play, it is your job to write a synopsis for that title.
Title: {title}
Playwright: This is a synopsis for the above play:"""
prompt_template = PromptTemplate(input_variables=["title"], template=template)
synopsis_chain = LLMChain(llm=llm, prompt=prompt_template, callbacks=callbacks)
test_prompts = [
    {"title": "documentary about good video games that push the boundary of game design"},
    {"title": "the phenomenon behind the remarkable speed of cheetahs"},
    {"title": "the best in class mlops tooling"},
]
synopsis_chain.apply(test_prompts)
aim_callback.flush_tracker(
    langchain_asset=synopsis_chain, experiment_name="scenario 3: Agent with Tools"
)


)
Scenario 3 The third scenario involves an agent with tools.
from langchain.agents import initialize_agent, load_tools
from langchain.agents import AgentType
# scenario 3 - Agent with Tools
tools = load_tools(["serpapi", "llm-math"], llm=llm, callbacks=callbacks)
agent = initialize_agent(
    tools,
    llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    callbacks=callbacks,
)
agent.run(
    "Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?"
)
aim_callback.flush_tracker(langchain_asset=agent, reset=False, finish=True)
> Entering new AgentExecutor chain...
 I need to find out who Leo DiCaprio's girlfriend is and then calculate her age raised to the 0.43 power.
Action: Search
Action Input: "Leo DiCaprio girlfriend"
Observation: Leonardo DiCaprio seemed to prove a long-held theory about his love life right after splitting from girlfriend Camila Morrone just months ...
Thought: I need to find out Camila Morrone's age
Action: Search
Action Input: "Camila Morrone age"
Observation: 25 years
Thought: I need to calculate 25 raised to the 0.43 power
Action: Calculator
Action Input: 25^0.43
Observation: Answer: 3.991298452658078
Thought: I now know the final answer
Final Answer: Camila Morrone is Leo DiCaprio's girlfriend and her current age raised to the 0.43 power is 3.991298452658078.
> Finished chain.
previous
AI21 Labs
next
AnalyticDB
By Harrison Chase

      © Copyright 2023, Harrison Chase.


AnalyticDB
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.md
.pdf
Tair
 Contents
Installation and Setup
Wrappers
VectorStore
Tair#
This page covers how to use the Tair ecosystem within LangChain.
Installation and Setup#
Install Tair Python SDK with pip install tair.
Wrappers#
VectorStore#
There exists a wrapper around TairVector, allowing you to use it as a vectorstore,
whether for semantic search or example selection.
To import this vectorstore:
from langchain.vectorstores import Tair
For a more detailed walkthrough of the Tair wrapper, see this notebook
previous
StochasticAI
next
Unstructured
 Contents

Installation and Setup
Wrappers
VectorStore
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.ipynb
.pdf
Chat Over Documents with Vectara
 Contents
Pass in chat history
Return Source Documents
ConversationalRetrievalChain with search_distance
ConversationalRetrievalChain with map_reduce
ConversationalRetrievalChain with Question Answering with sources
ConversationalRetrievalChain with streaming to stdout
get_chat_history Function
Chat Over Documents with Vectara#
This notebook is based on the chat_vector_db notebook, but using Vectara as the vector database.
import os
from langchain.vectorstores import Vectara
from langchain.vectorstores.vectara import VectaraRetriever
from langchain.llms import OpenAI
from langchain.chains import ConversationalRetrievalChain
Load in documents. You can replace this with a loader for whatever type of data you want
from langchain.document_loaders import TextLoader
loader = TextLoader("../../modules/state_of_the_union.txt")
documents = loader.load()
We now split the documents, create embeddings for them, and put them in a vectorstore. This allows us to do semantic search over them.
vectorstore = Vectara.from_documents(documents, embedding=None)
We can now create a memory object, which is neccessary to track the inputs/outputs and hold a conversation.
from langchain.memory import ConversationBufferMemory
memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
We now initialize the ConversationalRetrievalChain
openai_api_key = os.environ['OPENAI_API_KEY']
llm = OpenAI(openai_api_key=openai_api_key, temperature=0)
retriever = VectaraRetriever(vectorstore, alpha=0.025, k=5, filter=None)
print(type(vectorstore))
d = retriever.get_relevant_documents('What did the president say about Ketanji Brown Jackson')


qa = ConversationalRetrievalChain.from_llm(llm, retriever, memory=memory)
<class 'langchain.vectorstores.vectara.Vectara'>
query = "What did the president say about Ketanji Brown Jackson"
result = qa({"question": query})
result["answer"]
" The president said that Ketanji Brown Jackson is one of the nation's top legal minds, a former top litigator in private practice, and a former federal public defender."
query = "Did he mention who she suceeded"
result = qa({"question": query})
result['answer']
' Justice Stephen Breyer.'
Pass in chat history#
In the above example, we used a Memory object to track chat history. We can also just pass it in explicitly. In order to do this, we need to initialize a chain without any memory object.
qa = ConversationalRetrievalChain.from_llm(OpenAI(temperature=0), vectorstore.as_retriever())
Here’s an example of asking a question with no chat history
chat_history = []
query = "What did the president say about Ketanji Brown Jackson"
result = qa({"question": query, "chat_history": chat_history})
result["answer"]
" The president said that Ketanji Brown Jackson is one of the nation's top legal minds, a former top litigator in private practice, and a former federal public defender."
Here’s an example of asking a question with some chat history
chat_history = [(query, result["answer"])]
query = "Did he mention who she suceeded"
result = qa({"question": query, "chat_history": chat_history})
result['answer']
' Justice Stephen Breyer.'
Return Source Documents#


result['answer']
' Justice Stephen Breyer.'
Return Source Documents#
You can also easily return source documents from the ConversationalRetrievalChain. This is useful for when you want to inspect what documents were returned.
qa = ConversationalRetrievalChain.from_llm(llm, vectorstore.as_retriever(), return_source_documents=True)
chat_history = []
query = "What did the president say about Ketanji Brown Jackson"
result = qa({"question": query, "chat_history": chat_history})
result['source_documents'][0]
Document(page_content='Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence. A former top litigator in private practice. A former federal public defender.', metadata={'source': '../../modules/state_of_the_union.txt'})
ConversationalRetrievalChain with search_distance#
If you are using a vector store that supports filtering by search distance, you can add a threshold value parameter.
vectordbkwargs = {"search_distance": 0.9}
qa = ConversationalRetrievalChain.from_llm(OpenAI(temperature=0), vectorstore.as_retriever(), return_source_documents=True)
chat_history = []
query = "What did the president say about Ketanji Brown Jackson"
result = qa({"question": query, "chat_history": chat_history, "vectordbkwargs": vectordbkwargs})


ConversationalRetrievalChain with map_reduce#
We can also use different types of combine document chains with the ConversationalRetrievalChain chain.
from langchain.chains import LLMChain
from langchain.chains.question_answering import load_qa_chain
from langchain.chains.conversational_retrieval.prompts import CONDENSE_QUESTION_PROMPT
question_generator = LLMChain(llm=llm, prompt=CONDENSE_QUESTION_PROMPT)
doc_chain = load_qa_chain(llm, chain_type="map_reduce")
chain = ConversationalRetrievalChain(
    retriever=vectorstore.as_retriever(),
    question_generator=question_generator,
    combine_docs_chain=doc_chain,
)
chat_history = []
query = "What did the president say about Ketanji Brown Jackson"
result = chain({"question": query, "chat_history": chat_history})
result['answer']
' The president did not mention Ketanji Brown Jackson.'
ConversationalRetrievalChain with Question Answering with sources#
You can also use this chain with the question answering with sources chain.
from langchain.chains.qa_with_sources import load_qa_with_sources_chain
question_generator = LLMChain(llm=llm, prompt=CONDENSE_QUESTION_PROMPT)
doc_chain = load_qa_with_sources_chain(llm, chain_type="map_reduce")
chain = ConversationalRetrievalChain(
    retriever=vectorstore.as_retriever(),
    question_generator=question_generator,
    combine_docs_chain=doc_chain,
)
chat_history = []
query = "What did the president say about Ketanji Brown Jackson"
result = chain({"question": query, "chat_history": chat_history})
result['answer']


result = chain({"question": query, "chat_history": chat_history})
result['answer']
' The president did not mention Ketanji Brown Jackson.\nSOURCES: ../../modules/state_of_the_union.txt'
ConversationalRetrievalChain with streaming to stdout#
Output from the chain will be streamed to stdout token by token in this example.
from langchain.chains.llm import LLMChain
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain.chains.conversational_retrieval.prompts import CONDENSE_QUESTION_PROMPT, QA_PROMPT
from langchain.chains.question_answering import load_qa_chain
# Construct a ConversationalRetrievalChain with a streaming llm for combine docs
# and a separate, non-streaming llm for question generation
llm = OpenAI(temperature=0, openai_api_key=openai_api_key)
streaming_llm = OpenAI(streaming=True, callbacks=[StreamingStdOutCallbackHandler()], temperature=0, openai_api_key=openai_api_key)
question_generator = LLMChain(llm=llm, prompt=CONDENSE_QUESTION_PROMPT)
doc_chain = load_qa_chain(streaming_llm, chain_type="stuff", prompt=QA_PROMPT)
qa = ConversationalRetrievalChain(
    retriever=vectorstore.as_retriever(), combine_docs_chain=doc_chain, question_generator=question_generator)
chat_history = []
query = "What did the president say about Ketanji Brown Jackson"
result = qa({"question": query, "chat_history": chat_history})
 The president said that Ketanji Brown Jackson is one of the nation's top legal minds, a former top litigator in private practice, and a former federal public defender.
chat_history = [(query, result["answer"])]


chat_history = [(query, result["answer"])]
query = "Did he mention who she suceeded"
result = qa({"question": query, "chat_history": chat_history})
 Justice Stephen Breyer.
get_chat_history Function#
You can also specify a get_chat_history function, which can be used to format the chat_history string.
def get_chat_history(inputs) -> str:
    res = []
    for human, ai in inputs:
        res.append(f"Human:{human}\nAI:{ai}")
    return "\n".join(res)
qa = ConversationalRetrievalChain.from_llm(llm, vectorstore.as_retriever(), get_chat_history=get_chat_history)
chat_history = []
query = "What did the president say about Ketanji Brown Jackson"
result = qa({"question": query, "chat_history": chat_history})
result['answer']
" The president said that Ketanji Brown Jackson is one of the nation's top legal minds, a former top litigator in private practice, and a former federal public defender."
 Contents

Pass in chat history
Return Source Documents
ConversationalRetrievalChain with search_distance
ConversationalRetrievalChain with map_reduce
ConversationalRetrievalChain with Question Answering with sources
ConversationalRetrievalChain with streaming to stdout
get_chat_history Function
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.ipynb
.pdf
Vectara Text Generation
 Contents
Prepare Data
Set Up Vector DB
Set Up LLM Chain with Custom Prompt
Generate Text
Vectara Text Generation#
This notebook is based on chat_vector_db and adapted to Vectara.
Prepare Data#
First, we prepare the data. For this example, we fetch a documentation site that consists of markdown files hosted on Github and split them into small enough Documents.
from langchain.llms import OpenAI
from langchain.docstore.document import Document
import requests
from langchain.vectorstores import Vectara
from langchain.text_splitter import CharacterTextSplitter
from langchain.prompts import PromptTemplate
import pathlib
import subprocess
import tempfile
def get_github_docs(repo_owner, repo_name):
    with tempfile.TemporaryDirectory() as d:
        subprocess.check_call(
            f"git clone --depth 1 https://github.com/{repo_owner}/{repo_name}.git .",
            cwd=d,
            shell=True,
        )
        git_sha = (
            subprocess.check_output("git rev-parse HEAD", shell=True, cwd=d)
            .decode("utf-8")
            .strip()
        )
        repo_path = pathlib.Path(d)
        markdown_files = list(repo_path.glob("*/*.md")) + list(
            repo_path.glob("*/*.mdx")
        )
        for markdown_file in markdown_files:
            with open(markdown_file, "r") as f:
                relative_path = markdown_file.relative_to(repo_path)
                github_url = f"https://github.com/{repo_owner}/{repo_name}/blob/{git_sha}/{relative_path}"
                yield Document(page_content=f.read(), metadata={"source": github_url})
sources = get_github_docs("yirenlu92", "deno-manual-forked")
source_chunks = []


source_chunks = []
splitter = CharacterTextSplitter(separator=" ", chunk_size=1024, chunk_overlap=0)
for source in sources:
    for chunk in splitter.split_text(source.page_content):
        source_chunks.append(chunk)
Cloning into '.'...
Set Up Vector DB#
Now that we have the documentation content in chunks, let’s put all this information in a vector index for easy retrieval.
import os
search_index = Vectara.from_texts(source_chunks, embedding=None)
Set Up LLM Chain with Custom Prompt#
Next, let’s set up a simple LLM chain but give it a custom prompt for blog post generation. Note that the custom prompt is parameterized and takes two inputs: context, which will be the documents fetched from the vector search, and topic, which is given by the user.
from langchain.chains import LLMChain
prompt_template = """Use the context below to write a 400 word blog post about the topic below:
    Context: {context}
    Topic: {topic}
    Blog post:"""
PROMPT = PromptTemplate(
    template=prompt_template, input_variables=["context", "topic"]
)
llm = OpenAI(openai_api_key=os.environ['OPENAI_API_KEY'], temperature=0)
chain = LLMChain(llm=llm, prompt=PROMPT)
Generate Text#
Finally, we write a function to apply our inputs to the chain. The function takes an input parameter topic. We find the documents in the vector index that correspond to that topic, and use them as additional context in our simple LLM chain.
def generate_blog_post(topic):
    docs = search_index.similarity_search(topic, k=4)
    inputs = [{"context": doc.page_content, "topic": topic} for doc in docs]
    print(chain.apply(inputs))


print(chain.apply(inputs))
generate_blog_post("environment variables")


[{'text': '\n\nEnvironment variables are an essential part of any development workflow. They provide a way to store and access information that is specific to the environment in which the code is running. This can be especially useful when working with different versions of a language or framework, or when running code on different machines.\n\nThe Deno CLI tasks extension provides a way to easily manage environment variables when running Deno commands. This extension provides a task definition for allowing you to create tasks that execute the `deno` CLI from within the editor. The template for the Deno CLI tasks has the following interface, which can be configured in a `tasks.json` within your workspace:\n\nThe task definition includes the `type` field, which should be set to `deno`, and the `command` field, which is the `deno` command to run (e.g. `run`, `test`, `cache`, etc.). Additionally, you can specify additional arguments to pass on the command line, the current working directory to execute the command, and any environment variables.\n\nUsing environment variables with the


and any environment variables.\n\nUsing environment variables with the Deno CLI tasks extension is a great way to ensure that your code is running in the correct environment. For example, if you are running a test suite,'}, {'text': '\n\nEnvironment variables are an important part of any programming language, and they can be used to store and access data in a variety of ways. In this blog post, we\'ll be taking a look at environment variables specifically for the shell.\n\nShell variables are similar to environment variables, but they won\'t be exported to spawned commands. They are defined with the following syntax:\n\n```sh\nVAR_NAME=value\n```\n\nShell variables can be used to store and access data in a variety of ways. For example, you can use them to store values that you want to re-use, but don\'t want to be available in any spawned processes.\n\nFor example, if you wanted to store a value and then use it in a command, you could do something like this:\n\n```sh\nVAR=hello && echo $VAR && deno eval


&& echo $VAR && deno eval "console.log(\'Deno: \' + Deno.env.get(\'VAR\'))"\n```\n\nThis would output the following:\n\n```\nhello\nDeno: undefined\n```\n\nAs you can see, the value stored in the shell variable is not available in the spawned process.\n\n'}, {'text': '\n\nWhen it comes to developing applications, environment variables are an essential part of the process. Environment variables are used to store information that can be used by applications and scripts to customize their behavior. This is especially important when it comes to developing applications with Deno, as there are several environment variables that can impact the behavior of Deno.\n\nThe most important environment variable for Deno is `DENO_AUTH_TOKENS`. This environment variable is used to store authentication tokens that are used to access remote resources. This is especially important when it comes to accessing remote APIs or databases. Without the proper authentication tokens, Deno will not be able to access the remote resources.\n\nAnother important environment variable for Deno is `DENO_DIR`. This environment variable is used to


is `DENO_DIR`. This environment variable is used to store the directory where Deno will store its files. This includes the Deno executable, the Deno cache, and the Deno configuration files. By setting this environment variable, you can ensure that Deno will always be able to find the files it needs.\n\nFinally, there is the `DENO_PLUGINS` environment variable. This environment variable is used to store the list of plugins that Deno will use. This is important for customizing the'}, {'text': '\n\nEnvironment variables are a great way to store and access sensitive information in your Deno applications. Deno offers built-in support for environment variables with `Deno.env`, and you can also use a `.env` file to store and access environment variables. In this blog post, we\'ll explore both of these options and how to use them in your Deno applications.\n\n## Built-in `Deno.env`\n\nThe Deno runtime offers built-in support for environment variables with [`Deno.env`](https://deno.land/api@v1.25.3?s=Deno.env). `Deno.env` has getter and setter methods.


`Deno.env` has getter and setter methods. Here is example usage:\n\n```ts\nDeno.env.set("FIREBASE_API_KEY", "examplekey123");\nDeno.env.set("FIREBASE_AUTH_DOMAIN", "firebasedomain.com");\n\nconsole.log(Deno.env.get("FIREBASE_API_KEY")); // examplekey123\nconsole.log(Deno.env.get("FIREBASE_AUTH_'}]


Contents

Prepare Data
Set Up Vector DB
Set Up LLM Chain with Custom Prompt
Generate Text
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.md
.pdf
Autonomous Agents
 Contents
Baby AGI (Original Repo)
AutoGPT (Original Repo)
MetaPrompt (Original Repo)
Autonomous Agents#
Autonomous Agents are agents that designed to be more long running.
You give them one or multiple long term goals, and they independently execute towards those goals.
The applications combine tool usage and long term memory.
At the moment, Autonomous Agents are fairly experimental and based off of other open-source projects.
By implementing these open source projects in LangChain primitives we can get the benefits of LangChain -
easy switching and experimenting with multiple LLMs, usage of different vectorstores as memory,
usage of LangChain’s collection of tools.
Baby AGI (Original Repo)#
Baby AGI: a notebook implementing BabyAGI as LLM Chains
Baby AGI with Tools: building off the above notebook, this example substitutes in an agent with tools as the execution tools, allowing it to actually take actions.
AutoGPT (Original Repo)#
AutoGPT: a notebook implementing AutoGPT in LangChain primitives
WebSearch Research Assistant: a notebook showing how to use AutoGPT plus specific tools to act as research assistant that can use the web.
MetaPrompt (Original Repo)#
Meta-Prompt: a notebook implementing Meta-Prompt in LangChain primitives
previous
Callbacks
next
Agent Simulations
 Contents

Baby AGI (Original Repo)
AutoGPT (Original Repo)
MetaPrompt (Original Repo)
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.rst
.pdf
Evaluation
 Contents
The Problem
The Solution
The Examples
Other Examples
Evaluation#
Note
Conceptual Guide
This section of documentation covers how we approach and think about evaluation in LangChain.
Both evaluation of internal chains/agents, but also how we would recommend people building on top of LangChain approach evaluation.
The Problem#
It can be really hard to evaluate LangChain chains and agents.
There are two main reasons for this:
# 1: Lack of data
You generally don’t have a ton of data to evaluate your chains/agents over before starting a project.
This is usually because Large Language Models (the core of most chains/agents) are terrific few-shot and zero shot learners,
meaning you are almost always able to get started on a particular task (text-to-SQL, question answering, etc) without
a large dataset of examples.
This is in stark contrast to traditional machine learning where you had to first collect a bunch of datapoints
before even getting started using a model.
# 2: Lack of metrics
Most chains/agents are performing tasks for which there are not very good metrics to evaluate performance.
For example, one of the most common use cases is generating text of some form.
Evaluating generated text is much more complicated than evaluating a classification prediction, or a numeric prediction.
The Solution#
LangChain attempts to tackle both of those issues.
What we have so far are initial passes at solutions - we do not think we have a perfect solution.
So we very much welcome feedback, contributions, integrations, and thoughts on this.
Here is what we have for each problem so far:
# 1: Lack of data
We have started LangChainDatasets a Community space on Hugging Face.
We intend this to be a collection of open source datasets for evaluating common chains and agents.


We intend this to be a collection of open source datasets for evaluating common chains and agents.
We have contributed five datasets of our own to start, but we highly intend this to be a community effort.
In order to contribute a dataset, you simply need to join the community and then you will be able to upload datasets.
We’re also aiming to make it as easy as possible for people to create their own datasets.
As a first pass at this, we’ve added a QAGenerationChain, which given a document comes up
with question-answer pairs that can be used to evaluate question-answering tasks over that document down the line.
See this notebook for an example of how to use this chain.
# 2: Lack of metrics
We have two solutions to the lack of metrics.
The first solution is to use no metrics, and rather just rely on looking at results by eye to get a sense for how the chain/agent is performing.
To assist in this, we have developed (and will continue to develop) tracing, a UI-based visualizer of your chain and agent runs.
The second solution we recommend is to use Language Models themselves to evaluate outputs.
For this we have a few different chains and prompts aimed at tackling this issue.
The Examples#
We have created a bunch of examples combining the above two solutions to show how we internally evaluate chains and agents when we are developing.
In addition to the examples we’ve curated, we also highly welcome contributions here.
To facilitate that, we’ve included a template notebook for community members to use to build their own examples.
The existing examples we have are:
Question Answering (State of Union): A notebook showing evaluation of a question-answering task over a State-of-the-Union address.
Question Answering (Paul Graham Essay): A notebook showing evaluation of a question-answering task over a Paul Graham essay.


SQL Question Answering (Chinook): A notebook showing evaluation of a question-answering task over a SQL database (the Chinook database).
Agent Vectorstore: A notebook showing evaluation of an agent doing question answering while routing between two different vector databases.
Agent Search + Calculator: A notebook showing evaluation of an agent doing question answering using a Search engine and a Calculator as tools.
Evaluating an OpenAPI Chain: A notebook showing evaluation of an OpenAPI chain, including how to generate test data if you don’t have any.
Other Examples#
In addition, we also have some more generic resources for evaluation.
Question Answering: An overview of LLMs aimed at evaluating question answering systems in general.
Data Augmented Question Answering: An end-to-end example of evaluating a question answering system focused on a specific document (a RetrievalQAChain to be precise). This example highlights how to use LLMs to come up with question/answer examples to evaluate over, and then highlights how to use LLMs to evaluate performance on those generated examples.
Hugging Face Datasets: Covers an example of loading and using a dataset from Hugging Face for evaluation.
previous
Summarization
next
Agent Benchmarking: Search + Calculator
 Contents

The Problem
The Solution
The Examples
Other Examples
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.md
.pdf
Agents
 Contents
Create Your Own Agent
Step 1: Create Tools
(Optional) Step 2: Modify Agent
(Optional) Step 3: Modify Agent Executor
Examples
Agents#
Conceptual Guide
Agents can be used for a variety of tasks.
Agents combine the decision making ability of a language model with tools in order to create a system
that can execute and implement solutions on your behalf. Before reading any more, it is highly
recommended that you read the documentation in the agent module to understand the concepts associated with agents more.
Specifically, you should be familiar with what the agent, tool, and agent executor abstractions are before reading more.
Agent Documentation (for interacting with the outside world)
Create Your Own Agent#
Once you have read that documentation, you should be prepared to create your own agent.
What exactly does that involve?
Here’s how we recommend getting started with creating your own agent:
Step 1: Create Tools#
Agents are largely defined by the tools they can use.
If you have a specific task you want the agent to accomplish, you have to give it access to the right tools.
We have many tools natively in LangChain, so you should first look to see if any of them meet your needs.
But we also make it easy to define a custom tool, so if you need custom tools you should absolutely do that.
(Optional) Step 2: Modify Agent#
The built-in LangChain agent types are designed to work well in generic situations,
but you may be able to improve performance by modifying the agent implementation.
There are several ways you could do this:
Modify the base prompt. This can be used to give the agent more context on how it should behave, etc.
Modify the output parser. This is necessary if the agent is having trouble parsing the language model output.


Modify the output parser. This is necessary if the agent is having trouble parsing the language model output.
(Optional) Step 3: Modify Agent Executor#
This step is usually not necessary, as this is pretty general logic.
Possible reasons you would want to modify this include adding different stopping conditions, or handling errors
Examples#
Specific examples of agents include:
AI Plugins: an implementation of an agent that is designed to be able to use all AI Plugins.
Plug-and-PlAI (Plugins Database): an implementation of an agent that is designed to be able to use all AI Plugins retrieved from PlugNPlAI.
Wikibase Agent: an implementation of an agent that is designed to interact with Wikibase.
Sales GPT: This notebook demonstrates an implementation of a Context-Aware AI Sales agent.
Multi-Modal Output Agent: an implementation of a multi-modal output agent that can generate text and images.
previous
Agent Simulations
next
Question Answering over Docs
 Contents

Create Your Own Agent
Step 1: Create Tools
(Optional) Step 2: Modify Agent
(Optional) Step 3: Modify Agent Executor
Examples
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.md
.pdf
Extraction
Extraction#
Conceptual Guide
Most APIs and databases still deal with structured information.
Therefore, in order to better work with those, it can be useful to extract structured information from text.
Examples of this include:
Extracting a structured row to insert into a database from a sentence
Extracting multiple rows to insert into a database from a long document
Extracting the correct API parameters from a user query
This work is extremely related to output parsing.
Output parsers are responsible for instructing the LLM to respond in a specific format.
In this case, the output parsers specify the format of the data you would like to extract from the document.
Then, in addition to the output format instructions, the prompt should also contain the data you would like to extract information from.
While normal output parsers are good enough for basic structuring of response data,
when doing extraction you often want to extract more complicated or nested structures.
For a deep dive on extraction, we recommend checking out kor,
a library that uses the existing LangChain chain and OutputParser abstractions
but deep dives on allowing extraction of more complicated schemas.
previous
Interacting with APIs
next
Summarization
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.md
.pdf
Interacting with APIs
 Contents
Chains
Agents
Interacting with APIs#
Conceptual Guide
Lots of data and information is stored behind APIs.
This page covers all resources available in LangChain for working with APIs.
Chains#
If you are just getting started, and you have relatively simple apis, you should get started with chains.
Chains are a sequence of predetermined steps, so they are good to get started with as they give you more control and let you
understand what is happening better.
API Chain
Agents#
Agents are more complex, and involve multiple queries to the LLM to understand what to do.
The downside of agents are that you have less control. The upside is that they are more powerful,
which allows you to use them on larger and more complex schemas.
OpenAPI Agent
previous
Code Understanding
next
Extraction
 Contents

Chains
Agents
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.md
.pdf
Question Answering over Docs
 Contents
Document Question Answering
Adding in sources
Additional Related Resources
End-to-end examples
Question Answering over Docs#
Conceptual Guide
Question answering in this context refers to question answering over your document data.
For question answering over other types of data, please see other sources documentation like SQL database Question Answering or Interacting with APIs.
For question answering over many documents, you almost always want to create an index over the data.
This can be used to smartly access the most relevant documents for a given question, allowing you to avoid having to pass all the documents to the LLM (saving you time and money).
See this notebook for a more detailed introduction to this, but for a super quick start the steps involved are:
Load Your Documents
from langchain.document_loaders import TextLoader
loader = TextLoader('../state_of_the_union.txt')
See here for more information on how to get started with document loading.
Create Your Index
from langchain.indexes import VectorstoreIndexCreator
index = VectorstoreIndexCreator().from_loaders([loader])
The best and most popular index by far at the moment is the VectorStore index.
Query Your Index
query = "What did the president say about Ketanji Brown Jackson"
index.query(query)
Alternatively, use query_with_sources to also get back the sources involved
query = "What did the president say about Ketanji Brown Jackson"
index.query_with_sources(query)
Again, these high level interfaces obfuscate a lot of what is going on under the hood, so please see this notebook for a lower level walkthrough.
Document Question Answering#
Question answering involves fetching multiple documents, and then asking a question of them.
The LLM response will contain the answer to your question, based on the content of the documents.


The LLM response will contain the answer to your question, based on the content of the documents.
The recommended way to get started using a question answering chain is:
from langchain.chains.question_answering import load_qa_chain
chain = load_qa_chain(llm, chain_type="stuff")
chain.run(input_documents=docs, question=query)
The following resources exist:
Question Answering Notebook: A notebook walking through how to accomplish this task.
VectorDB Question Answering Notebook: A notebook walking through how to do question answering over a vector database. This can often be useful for when you have a LOT of documents, and you don’t want to pass them all to the LLM, but rather first want to do some semantic search over embeddings.
Adding in sources#
There is also a variant of this, where in addition to responding with the answer the language model will also cite its sources (eg which of the documents passed in it used).
The recommended way to get started using a question answering with sources chain is:
from langchain.chains.qa_with_sources import load_qa_with_sources_chain
chain = load_qa_with_sources_chain(llm, chain_type="stuff")
chain({"input_documents": docs, "question": query}, return_only_outputs=True)
The following resources exist:
QA With Sources Notebook: A notebook walking through how to accomplish this task.
VectorDB QA With Sources Notebook: A notebook walking through how to do question answering with sources over a vector database. This can often be useful for when you have a LOT of documents, and you don’t want to pass them all to the LLM, but rather first want to do some semantic search over embeddings.
Additional Related Resources#
Additional related resources include:


Additional Related Resources#
Additional related resources include:
Utilities for working with Documents: Guides on how to use several of the utilities which will prove helpful for this task, including Text Splitters (for splitting up long documents) and Embeddings & Vectorstores (useful for the above Vector DB example).
CombineDocuments Chains: A conceptual overview of specific types of chains by which you can accomplish this task.
End-to-end examples#
For examples to this done in an end-to-end manner, please see the following resources:
Semantic search over a group chat with Sources Notebook: A notebook that semantically searches over a group chat conversation.
previous
Agents
next
Chatbots
 Contents

Document Question Answering
Adding in sources
Additional Related Resources
End-to-end examples
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.md
.pdf
Chatbots
Chatbots#
Conceptual Guide
Since language models are good at producing text, that makes them ideal for creating chatbots.
Aside from the base prompts/LLMs, an important concept to know for Chatbots is memory.
Most chat based applications rely on remembering what happened in previous interactions, which memory is designed to help with.
The following resources exist:
ChatGPT Clone: A notebook walking through how to recreate a ChatGPT-like experience with LangChain.
Conversation Memory: A notebook walking through how to use different types of conversational memory.
Conversation Agent: A notebook walking through how to create an agent optimized for conversation.
Additional related resources include:
Memory Key Concepts: Explanation of key concepts related to memory.
Memory Examples: A collection of how-to examples for working with memory.
More end-to-end examples include:
Voice Assistant: A notebook walking through how to create a voice assistant using LangChain.
previous
Question Answering over Docs
next
Querying Tabular Data
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.md
.pdf
Agent Simulations
 Contents
Simulations with One Agent
Simulations with Two Agents
Simulations with Multiple Agents
Agent Simulations#
Agent simulations involve interacting one of more agents with each other.
Agent simulations generally involve two main components:
Long Term Memory
Simulation Environment
Specific implementations of agent simulations (or parts of agent simulations) include:
Simulations with One Agent#
Simulated Environment: Gymnasium: an example of how to create a simple agent-environment interaction loop with Gymnasium (formerly OpenAI Gym).
Simulations with Two Agents#
CAMEL: an implementation of the CAMEL (Communicative Agents for “Mind” Exploration of Large Scale Language Model Society) paper, where two agents communicate with each other.
Two Player D&D: an example of how to use a generic simulator for two agents to implement a variant of the popular Dungeons & Dragons role playing game.
Agent Debates with Tools: an example of how to enable Dialogue Agents to use tools to inform their responses.
Simulations with Multiple Agents#
Multi-Player D&D: an example of how to use a generic dialogue simulator for multiple dialogue agents with a custom speaker-ordering, illustrated with a variant of the popular Dungeons & Dragons role playing game.
Decentralized Speaker Selection: an example of how to implement a multi-agent dialogue without a fixed schedule for who speaks when. Instead the agents decide for themselves who speaks by outputting bids to speak. This example shows how to do this in the context of a fictitious presidential debate.
Authoritarian Speaker Selection: an example of how to implement a multi-agent dialogue, where a privileged agent directs who speaks what. This example also showcases how to enable the privileged agent to determine when the conversation terminates. This example shows how to do this in the context of a fictitious news show.


Simulated Environment: PettingZoo: an example of how to create a agent-environment interaction loop for multiple agents with PettingZoo (a multi-agent version of Gymnasium).
Generative Agents: This notebook implements a generative agent based on the paper Generative Agents: Interactive Simulacra of Human Behavior by Park, et. al.
previous
Autonomous Agents
next
Agents
 Contents

Simulations with One Agent
Simulations with Two Agents
Simulations with Multiple Agents
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.md
.pdf
Code Understanding
 Contents
Conversational Retriever Chain
Code Understanding#
Overview
LangChain is a useful tool designed to parse GitHub code repositories. By leveraging VectorStores, Conversational RetrieverChain, and GPT-4, it can answer questions in the context of an entire GitHub repository or generate new code. This documentation page outlines the essential components of the system and guides using LangChain for better code comprehension, contextual question answering, and code generation in GitHub repositories.
Conversational Retriever Chain#
Conversational RetrieverChain is a retrieval-focused system that interacts with the data stored in a VectorStore. Utilizing advanced techniques, like context-aware filtering and ranking, it retrieves the most relevant code snippets and information for a given user query. Conversational RetrieverChain is engineered to deliver high-quality, pertinent results while considering conversation history and context.
LangChain Workflow for Code Understanding and Generation
Index the code base: Clone the target repository, load all files within, chunk the files, and execute the indexing process. Optionally, you can skip this step and use an already indexed dataset.
Embedding and Code Store: Code snippets are embedded using a code-aware embedding model and stored in a VectorStore.
Query Understanding: GPT-4 processes user queries, grasping the context and extracting relevant details.
Construct the Retriever: Conversational RetrieverChain searches the VectorStore to identify the most relevant code snippets for a given query.
Build the Conversational Chain: Customize the retriever settings and define any user-defined filters as needed.
Ask questions: Define a list of questions to ask about the codebase, and then use the ConversationalRetrievalChain to generate context-aware answers. The LLM (GPT-4) generates comprehensive, context-aware answers based on retrieved code snippets and conversation history.
The full tutorial is available below.


The full tutorial is available below.
Twitter the-algorithm codebase analysis with Deep Lake: A notebook walking through how to parse github source code and run queries conversation.
LangChain codebase analysis with Deep Lake: A notebook walking through how to analyze and do question answering over THIS code base.
previous
Querying Tabular Data
next
Interacting with APIs
 Contents

Conversational Retriever Chain
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.md
.pdf
Summarization
Summarization#
Conceptual Guide
Summarization involves creating a smaller summary of multiple longer documents.
This can be useful for distilling long documents into the core pieces of information.
The recommended way to get started using a summarization chain is:
from langchain.chains.summarize import load_summarize_chain
chain = load_summarize_chain(llm, chain_type="map_reduce")
chain.run(docs)
The following resources exist:
Summarization Notebook: A notebook walking through how to accomplish this task.
Additional related resources include:
Utilities for working with Documents: Guides on how to use several of the utilities which will prove helpful for this task, including Text Splitters (for splitting up long documents).
previous
Extraction
next
Evaluation
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.md
.pdf
Querying Tabular Data
 Contents
Document Loading
Querying
Chains
Agents
Querying Tabular Data#
Conceptual Guide
Lots of data and information is stored in tabular data, whether it be csvs, excel sheets, or SQL tables.
This page covers all resources available in LangChain for working with data in this format.
Document Loading#
If you have text data stored in a tabular format, you may want to load the data into a Document and then index it as you would
other text/unstructured data. For this, you should use a document loader like the CSVLoader
and then you should create an index over that data, and query it that way.
Querying#
If you have more numeric tabular data, or have a large amount of data and don’t want to index it, you should get started
by looking at various chains and agents we have for dealing with this data.
Chains#
If you are just getting started, and you have relatively small/simple tabular data, you should get started with chains.
Chains are a sequence of predetermined steps, so they are good to get started with as they give you more control and let you
understand what is happening better.
SQL Database Chain
Agents#
Agents are more complex, and involve multiple queries to the LLM to understand what to do.
The downside of agents are that you have less control. The upside is that they are more powerful,
which allows you to use them on larger databases and more complex schemas.
SQL Agent
Pandas Agent
CSV Agent
previous
Chatbots
next
Code Understanding
 Contents

Document Loading
Querying
Chains
Agents
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 28, 2023.


.ipynb
.pdf
AutoGPT
 Contents
Set up tools
Set up memory
Setup model and AutoGPT
Run an example
AutoGPT#
Implementation of https://github.com/Significant-Gravitas/Auto-GPT but with LangChain primitives (LLMs, PromptTemplates, VectorStores, Embeddings, Tools)
Set up tools#
We’ll set up an AutoGPT with a search tool, and write-file tool, and a read-file tool
from langchain.utilities import SerpAPIWrapper
from langchain.agents import Tool
from langchain.tools.file_management.write import WriteFileTool
from langchain.tools.file_management.read import ReadFileTool
search = SerpAPIWrapper()
tools = [
    Tool(
        name = "search",
        func=search.run,
        description="useful for when you need to answer questions about current events. You should ask targeted questions"
    ),
    WriteFileTool(),
    ReadFileTool(),
]
Set up memory#
The memory here is used for the agents intermediate steps
from langchain.vectorstores import FAISS
from langchain.docstore import InMemoryDocstore
from langchain.embeddings import OpenAIEmbeddings
# Define your embedding model
embeddings_model = OpenAIEmbeddings()
# Initialize the vectorstore as empty
import faiss
embedding_size = 1536
index = faiss.IndexFlatL2(embedding_size)
vectorstore = FAISS(embeddings_model.embed_query, index, InMemoryDocstore({}), {})
Setup model and AutoGPT#
Initialize everything! We will use ChatOpenAI model
from langchain.experimental import AutoGPT
from langchain.chat_models import ChatOpenAI
agent = AutoGPT.from_llm_and_tools(
    ai_name="Tom",
    ai_role="Assistant",
    tools=tools,


ai_name="Tom",
    ai_role="Assistant",
    tools=tools,
    llm=ChatOpenAI(temperature=0),
    memory=vectorstore.as_retriever()
)
# Set verbose to be true
agent.chain.verbose = True
Run an example#
Here we will make it write a weather report for SF
agent.run(["write a weather report for SF today"])
> Entering new LLMChain chain...
Prompt after formatting:
System: You are Tom, Assistant
Your decisions must always be made independently
            without seeking user assistance. Play to your strengths
            as an LLM and pursue simple strategies with no legal complications.
            If you have completed all your tasks,
            make sure to use the "finish" command.
GOALS:
1. write a weather report for SF today
Constraints:
1. ~4000 word limit for short term memory. Your short term memory is short, so immediately save important information to files.
2. If you are unsure how you previously did something or want to recall past events, thinking about similar events will help you remember.
3. No user assistance
4. Exclusively use the commands listed in double quotes e.g. "command name"
Commands:
1. search: useful for when you need to answer questions about current events. You should ask targeted questions, args json schema: {"query": {"title": "Query", "type": "string"}}
2. write_file: Write file to disk, args json schema: {"file_path": {"title": "File Path", "description": "name of file", "type": "string"}, "text": {"title": "Text", "description": "text to write to file", "type": "string"}}


3. read_file: Read file from disk, args json schema: {"file_path": {"title": "File Path", "description": "name of file", "type": "string"}}
4. finish: use this to signal that you have finished all your objectives, args: "response": "final response to let people know you have finished your objectives"
Resources:
1. Internet access for searches and information gathering.
2. Long Term memory management.
3. GPT-3.5 powered Agents for delegation of simple tasks.
4. File output.
Performance Evaluation:
1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities.
2. Constructively self-criticize your big-picture behavior constantly.
3. Reflect on past decisions and strategies to refine your approach.
4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.
You should only respond in JSON format as described below
Response Format:
{
    "thoughts": {
        "text": "thought",
        "reasoning": "reasoning",
        "plan": "- short bulleted\n- list that conveys\n- long-term plan",
        "criticism": "constructive self-criticism",
        "speak": "thoughts summary to say to user"
    },
    "command": {
        "name": "command name",
        "args": {
            "arg name": "value"
        }
    }
}
Ensure the response can be parsed by Python json.loads
System: The current time and date is Tue Apr 18 21:31:28 2023
System: This reminds you of these events from your past:
[]


System: This reminds you of these events from your past:
[]
Human: Determine which next command to use, and respond using the format specified above:
> Finished chain.
{
    "thoughts": {
        "text": "I will start by writing a weather report for San Francisco today. I will use the 'search' command to find the current weather conditions.",
        "reasoning": "I need to gather information about the current weather conditions in San Francisco to write an accurate weather report.",
        "plan": "- Use the 'search' command to find the current weather conditions in San Francisco\n- Write a weather report based on the information gathered",
        "criticism": "I need to make sure that the information I gather is accurate and up-to-date.",
        "speak": "I will use the 'search' command to find the current weather conditions in San Francisco."
    },
    "command": {
        "name": "search",
        "args": {
            "query": "what is the current weather in san francisco"
        }
    }
}
> Entering new LLMChain chain...
Prompt after formatting:
System: You are Tom, Assistant
Your decisions must always be made independently
            without seeking user assistance. Play to your strengths
            as an LLM and pursue simple strategies with no legal complications.
            If you have completed all your tasks,
            make sure to use the "finish" command.
GOALS:
1. write a weather report for SF today
Constraints:
1. ~4000 word limit for short term memory. Your short term memory is short, so immediately save important information to files.
2. If you are unsure how you previously did something or want to recall past events, thinking about similar events will help you remember.
3. No user assistance


3. No user assistance
4. Exclusively use the commands listed in double quotes e.g. "command name"
Commands:
1. search: useful for when you need to answer questions about current events. You should ask targeted questions, args json schema: {"query": {"title": "Query", "type": "string"}}
2. write_file: Write file to disk, args json schema: {"file_path": {"title": "File Path", "description": "name of file", "type": "string"}, "text": {"title": "Text", "description": "text to write to file", "type": "string"}}
3. read_file: Read file from disk, args json schema: {"file_path": {"title": "File Path", "description": "name of file", "type": "string"}}
4. finish: use this to signal that you have finished all your objectives, args: "response": "final response to let people know you have finished your objectives"
Resources:
1. Internet access for searches and information gathering.
2. Long Term memory management.
3. GPT-3.5 powered Agents for delegation of simple tasks.
4. File output.
Performance Evaluation:
1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities.
2. Constructively self-criticize your big-picture behavior constantly.
3. Reflect on past decisions and strategies to refine your approach.
4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.
You should only respond in JSON format as described below
Response Format:
{
    "thoughts": {
        "text": "thought",
        "reasoning": "reasoning",


"text": "thought",
        "reasoning": "reasoning",
        "plan": "- short bulleted\n- list that conveys\n- long-term plan",
        "criticism": "constructive self-criticism",
        "speak": "thoughts summary to say to user"
    },
    "command": {
        "name": "command name",
        "args": {
            "arg name": "value"
        }
    }
}
Ensure the response can be parsed by Python json.loads
System: The current time and date is Tue Apr 18 21:31:39 2023
System: This reminds you of these events from your past:


System: This reminds you of these events from your past:
['Assistant Reply: {\n    "thoughts": {\n        "text": "I will start by writing a weather report for San Francisco today. I will use the \'search\' command to find the current weather conditions.",\n        "reasoning": "I need to gather information about the current weather conditions in San Francisco to write an accurate weather report.",\n        "plan": "- Use the \'search\' command to find the current weather conditions in San Francisco\\n- Write a weather report based on the information gathered",\n        "criticism": "I need to make sure that the information I gather is accurate and up-to-date.",\n        "speak": "I will use the \'search\' command to find the current weather conditions in San Francisco."\n    },\n    "command": {\n        "name": "search",\n        "args": {\n            "query": "what is the current weather in san francisco"\n        }\n    }\n} \nResult: Command search returned: Current Weather ; 54°F · Sunny ; RealFeel® 66°. Pleasant. RealFeel Guide. Pleasant. 63° to 81°. Most consider this temperature range ideal. LEARN MORE. RealFeel ... ']
Human: Determine which next command to use, and respond using the format specified above:
AI: {
    "thoughts": {
        "text": "I will start by writing a weather report for San Francisco today. I will use the 'search' command to find the current weather conditions.",
        "reasoning": "I need to gather information about the current weather conditions in San Francisco to write an accurate weather report.",
        "plan": "- Use the 'search' command to find the current weather conditions in San Francisco\n- Write a weather report based on the information gathered",