.ipynb
.pdf
Example Selectors
 Contents
LengthBased ExampleSelector
Similarity ExampleSelector
Maximal Marginal Relevance ExampleSelector
NGram Overlap ExampleSelector
Example Selectors#
If you have a large number of examples, you may need to select which ones to include in the prompt. The ExampleSelector is the class responsible for doing so. The base interface is defined as below.
class BaseExampleSelector(ABC):
    """Interface for selecting examples to include in prompts."""
    @abstractmethod
    def select_examples(self, input_variables: Dict[str, str]) -> List[dict]:
        """Select which examples to use based on the inputs."""
The only method it needs to expose is a select_examples method. This takes in the input variables and then returns a list of examples. It is up to each specific implementation as to how those examples are selected. Let’s take a look at some below.
from langchain.prompts import FewShotPromptTemplate
LengthBased ExampleSelector#
This ExampleSelector selects which examples to use based on length. This is useful when you are worried about constructing a prompt that will go over the length of the context window. For longer inputs, it will select fewer examples to include, while for shorter inputs it will select more.
from langchain.prompts import PromptTemplate
from langchain.prompts.example_selector import LengthBasedExampleSelector
# These are a lot of examples of a pretend task of creating antonyms.
examples = [
    {"input": "happy", "output": "sad"},
    {"input": "tall", "output": "short"},
    {"input": "energetic", "output": "lethargic"},
    {"input": "sunny", "output": "gloomy"},
    {"input": "windy", "output": "calm"},
]
example_prompt = PromptTemplate(


]
example_prompt = PromptTemplate(
    input_variables=["input", "output"],
    template="Input: {input}\nOutput: {output}",
)
example_selector = LengthBasedExampleSelector(
    # These are the examples it has available to choose from.
    examples=examples,
    # This is the PromptTemplate being used to format the examples.
    example_prompt=example_prompt,
    # This is the maximum length that the formatted examples should be.
    # Length is measured by the get_text_length function below.
    max_length=25,
    # This is the function used to get the length of a string, which is used
    # to determine which examples to include. It is commented out because
    # it is provided as a default value if none is specified.
    # get_text_length: Callable[[str], int] = lambda x: len(re.split("\n| ", x))
)
dynamic_prompt = FewShotPromptTemplate(
    # We provide an ExampleSelector instead of examples.
    example_selector=example_selector,
    example_prompt=example_prompt,
    prefix="Give the antonym of every input",
    suffix="Input: {adjective}\nOutput:",
    input_variables=["adjective"],
)
# An example with small input, so it selects all examples.
print(dynamic_prompt.format(adjective="big"))
Give the antonym of every input
Input: happy
Output: sad
Input: tall
Output: short
Input: energetic
Output: lethargic
Input: sunny
Output: gloomy
Input: windy
Output: calm
Input: big
Output:
# An example with long input, so it selects only one example.
long_string = "big and huge and massive and large and gigantic and tall and much much much much much bigger than everything else"


print(dynamic_prompt.format(adjective=long_string))
Give the antonym of every input
Input: happy
Output: sad
Input: big and huge and massive and large and gigantic and tall and much much much much much bigger than everything else
Output:
# You can add an example to an example selector as well.
new_example = {"input": "big", "output": "small"}
dynamic_prompt.example_selector.add_example(new_example)
print(dynamic_prompt.format(adjective="enthusiastic"))
Give the antonym of every input
Input: happy
Output: sad
Input: tall
Output: short
Input: energetic
Output: lethargic
Input: sunny
Output: gloomy
Input: windy
Output: calm
Input: big
Output: small
Input: enthusiastic
Output:
Similarity ExampleSelector#
The SemanticSimilarityExampleSelector selects examples based on which examples are most similar to the inputs. It does this by finding the examples with the embeddings that have the greatest cosine similarity with the inputs.
from langchain.prompts.example_selector import SemanticSimilarityExampleSelector
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
example_selector = SemanticSimilarityExampleSelector.from_examples(
    # This is the list of examples available to select from.
    examples,
    # This is the embedding class used to produce embeddings which are used to measure semantic similarity.
    OpenAIEmbeddings(),
    # This is the VectorStore class that is used to store the embeddings and do a similarity search over.
    Chroma,
    # This is the number of examples to produce.
    k=1
)
similar_prompt = FewShotPromptTemplate(
    # We provide an ExampleSelector instead of examples.
    example_selector=example_selector,
    example_prompt=example_prompt,


example_selector=example_selector,
    example_prompt=example_prompt,
    prefix="Give the antonym of every input",
    suffix="Input: {adjective}\nOutput:",
    input_variables=["adjective"],
)
Running Chroma using direct local API.
Using DuckDB in-memory for database. Data will be transient.
# Input is a feeling, so should select the happy/sad example
print(similar_prompt.format(adjective="worried"))
Give the antonym of every input
Input: happy
Output: sad
Input: worried
Output:
# Input is a measurement, so should select the tall/short example
print(similar_prompt.format(adjective="fat"))
Give the antonym of every input
Input: happy
Output: sad
Input: fat
Output:
# You can add new examples to the SemanticSimilarityExampleSelector as well
similar_prompt.example_selector.add_example({"input": "enthusiastic", "output": "apathetic"})
print(similar_prompt.format(adjective="joyful"))
Give the antonym of every input
Input: happy
Output: sad
Input: joyful
Output:
Maximal Marginal Relevance ExampleSelector#
The MaxMarginalRelevanceExampleSelector selects examples based on a combination of which examples are most similar to the inputs, while also optimizing for diversity. It does this by finding the examples with the embeddings that have the greatest cosine similarity with the inputs, and then iteratively adding them while penalizing them for closeness to already selected examples.
from langchain.prompts.example_selector import MaxMarginalRelevanceExampleSelector
from langchain.vectorstores import FAISS
example_selector = MaxMarginalRelevanceExampleSelector.from_examples(
    # This is the list of examples available to select from.
    examples,


# This is the list of examples available to select from.
    examples,
    # This is the embedding class used to produce embeddings which are used to measure semantic similarity.
    OpenAIEmbeddings(),
    # This is the VectorStore class that is used to store the embeddings and do a similarity search over.
    FAISS,
    # This is the number of examples to produce.
    k=2
)
mmr_prompt = FewShotPromptTemplate(
    # We provide an ExampleSelector instead of examples.
    example_selector=example_selector,
    example_prompt=example_prompt,
    prefix="Give the antonym of every input",
    suffix="Input: {adjective}\nOutput:",
    input_variables=["adjective"],
)
# Input is a feeling, so should select the happy/sad example as the first one
print(mmr_prompt.format(adjective="worried"))
Give the antonym of every input
Input: happy
Output: sad
Input: windy
Output: calm
Input: worried
Output:
# Let's compare this to what we would just get if we went solely off of similarity
similar_prompt.example_selector.k = 2
print(similar_prompt.format(adjective="worried"))
Give the antonym of every input
Input: enthusiastic
Output: apathetic
Input: worried
Output:
NGram Overlap ExampleSelector#
The NGramOverlapExampleSelector selects and orders examples based on which examples are most similar to the input, according to an ngram overlap score. The ngram overlap score is a float between 0.0 and 1.0, inclusive.


The selector allows for a threshold score to be set. Examples with an ngram overlap score less than or equal to the threshold are excluded. The threshold is set to -1.0, by default, so will not exclude any examples, only reorder them. Setting the threshold to 0.0 will exclude examples that have no ngram overlaps with the input.
from langchain.prompts import PromptTemplate
from langchain.prompts.example_selector.ngram_overlap import NGramOverlapExampleSelector
# These are examples of a fictional translation task.
examples = [
    {"input": "See Spot run.", "output": "Ver correr a Spot."},
    {"input": "My dog barks.", "output": "Mi perro ladra."},
    {"input": "Spot can run.", "output": "Spot puede correr."},
]
example_prompt = PromptTemplate(
    input_variables=["input", "output"],
    template="Input: {input}\nOutput: {output}",
)
example_selector = NGramOverlapExampleSelector(
    # These are the examples it has available to choose from.
    examples=examples,
    # This is the PromptTemplate being used to format the examples.
    example_prompt=example_prompt,
    # This is the threshold, at which selector stops.
    # It is set to -1.0 by default.
    threshold=-1.0,
    # For negative threshold:
    # Selector sorts examples by ngram overlap score, and excludes none.
    # For threshold greater than 1.0:
    # Selector excludes all examples, and returns an empty list.
    # For threshold equal to 0.0:
    # Selector sorts examples by ngram overlap score,
    # and excludes those with no ngram overlap with input.
)


# and excludes those with no ngram overlap with input.
)
dynamic_prompt = FewShotPromptTemplate(
    # We provide an ExampleSelector instead of examples.
    example_selector=example_selector,
    example_prompt=example_prompt,
    prefix="Give the Spanish translation of every input",
    suffix="Input: {sentence}\nOutput:",
    input_variables=["sentence"],
)
# An example input with large ngram overlap with "Spot can run."
# and no overlap with "My dog barks."
print(dynamic_prompt.format(sentence="Spot can run fast."))
Give the Spanish translation of every input
Input: Spot can run.
Output: Spot puede correr.
Input: See Spot run.
Output: Ver correr a Spot.
Input: My dog barks.
Output: Mi perro ladra.
Input: Spot can run fast.
Output:
# You can add examples to NGramOverlapExampleSelector as well.
new_example = {"input": "Spot plays fetch.", "output": "Spot juega a buscar."}
example_selector.add_example(new_example)
print(dynamic_prompt.format(sentence="Spot can run fast."))
Give the Spanish translation of every input
Input: Spot can run.
Output: Spot puede correr.
Input: See Spot run.
Output: Ver correr a Spot.
Input: Spot plays fetch.
Output: Spot juega a buscar.
Input: My dog barks.
Output: Mi perro ladra.
Input: Spot can run fast.
Output:
# You can set a threshold at which examples are excluded.
# For example, setting threshold equal to 0.0
# excludes examples with no ngram overlaps with input.
# Since "My dog barks." has no ngram overlaps with "Spot can run fast."


# Since "My dog barks." has no ngram overlaps with "Spot can run fast."
# it is excluded.
example_selector.threshold=0.0
print(dynamic_prompt.format(sentence="Spot can run fast."))
Give the Spanish translation of every input
Input: Spot can run.
Output: Spot puede correr.
Input: See Spot run.
Output: Ver correr a Spot.
Input: Spot plays fetch.
Output: Spot juega a buscar.
Input: Spot can run fast.
Output:
# Setting small nonzero threshold
example_selector.threshold=0.09
print(dynamic_prompt.format(sentence="Spot can play fetch."))
Give the Spanish translation of every input
Input: Spot can run.
Output: Spot puede correr.
Input: Spot plays fetch.
Output: Spot juega a buscar.
Input: Spot can play fetch.
Output:
# Setting threshold greater than 1.0
example_selector.threshold=1.0+1e-9
print(dynamic_prompt.format(sentence="Spot can play fetch."))
Give the Spanish translation of every input
Input: Spot can play fetch.
Output:
previous
Prompt Serialization
next
Output Parsers
 Contents

LengthBased ExampleSelector
Similarity ExampleSelector
Maximal Marginal Relevance ExampleSelector
NGram Overlap ExampleSelector
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
Provide few shot examples to a prompt
 Contents
Use Case
Using an example set
Create the example set
Create a formatter for the few shot examples
Feed examples and formatter to FewShotPromptTemplate
Using an example selector
Feed examples into ExampleSelector
Feed example selector into FewShotPromptTemplate
Provide few shot examples to a prompt#
In this tutorial, we’ll learn how to create a prompt template that uses few shot examples.
We’ll use the FewShotPromptTemplate class to create a prompt template that uses few shot examples. This class either takes in a set of examples, or an ExampleSelector object. In this tutorial, we’ll go over both options.
Use Case#
In this tutorial, we’ll configure few shot examples for self-ask with search.
Using an example set#
Create the example set#
To get started, create a list of few shot examples. Each example should be a dictionary with the keys being the input variables and the values being the values for those input variables.
from langchain.prompts.few_shot import FewShotPromptTemplate
from langchain.prompts.prompt import PromptTemplate
examples = [
  {
    "question": "Who lived longer, Muhammad Ali or Alan Turing?",
    "answer":
"""
Are follow up questions needed here: Yes.
Follow up: How old was Muhammad Ali when he died?
Intermediate answer: Muhammad Ali was 74 years old when he died.
Follow up: How old was Alan Turing when he died?
Intermediate answer: Alan Turing was 41 years old when he died.
So the final answer is: Muhammad Ali
"""
  },
  {
    "question": "When was the founder of craigslist born?",
    "answer":
"""
Are follow up questions needed here: Yes.
Follow up: Who was the founder of craigslist?


Follow up: Who was the founder of craigslist?
Intermediate answer: Craigslist was founded by Craig Newmark.
Follow up: When was Craig Newmark born?
Intermediate answer: Craig Newmark was born on December 6, 1952.
So the final answer is: December 6, 1952
"""
  },
  {
    "question": "Who was the maternal grandfather of George Washington?",
    "answer":
"""
Are follow up questions needed here: Yes.
Follow up: Who was the mother of George Washington?
Intermediate answer: The mother of George Washington was Mary Ball Washington.
Follow up: Who was the father of Mary Ball Washington?
Intermediate answer: The father of Mary Ball Washington was Joseph Ball.
So the final answer is: Joseph Ball
"""
  },
  {
    "question": "Are both the directors of Jaws and Casino Royale from the same country?",
    "answer":
"""
Are follow up questions needed here: Yes.
Follow up: Who is the director of Jaws?
Intermediate Answer: The director of Jaws is Steven Spielberg.
Follow up: Where is Steven Spielberg from?
Intermediate Answer: The United States.
Follow up: Who is the director of Casino Royale?
Intermediate Answer: The director of Casino Royale is Martin Campbell.
Follow up: Where is Martin Campbell from?
Intermediate Answer: New Zealand.
So the final answer is: No
"""
  }
]
Create a formatter for the few shot examples#
Configure a formatter that will format the few shot examples into a string. This formatter should be a PromptTemplate object.
example_prompt = PromptTemplate(input_variables=["question", "answer"], template="Question: {question}\n{answer}")
print(example_prompt.format(**examples[0]))
Question: Who lived longer, Muhammad Ali or Alan Turing?


Question: Who lived longer, Muhammad Ali or Alan Turing?
Are follow up questions needed here: Yes.
Follow up: How old was Muhammad Ali when he died?
Intermediate answer: Muhammad Ali was 74 years old when he died.
Follow up: How old was Alan Turing when he died?
Intermediate answer: Alan Turing was 41 years old when he died.
So the final answer is: Muhammad Ali
Feed examples and formatter to FewShotPromptTemplate#
Finally, create a FewShotPromptTemplate object. This object takes in the few shot examples and the formatter for the few shot examples.
prompt = FewShotPromptTemplate(
    examples=examples,
    example_prompt=example_prompt,
    suffix="Question: {input}",
    input_variables=["input"]
)
print(prompt.format(input="Who was the father of Mary Ball Washington?"))
Question: Who lived longer, Muhammad Ali or Alan Turing?
Are follow up questions needed here: Yes.
Follow up: How old was Muhammad Ali when he died?
Intermediate answer: Muhammad Ali was 74 years old when he died.
Follow up: How old was Alan Turing when he died?
Intermediate answer: Alan Turing was 41 years old when he died.
So the final answer is: Muhammad Ali
Question: When was the founder of craigslist born?
Are follow up questions needed here: Yes.
Follow up: Who was the founder of craigslist?
Intermediate answer: Craigslist was founded by Craig Newmark.
Follow up: When was Craig Newmark born?
Intermediate answer: Craig Newmark was born on December 6, 1952.
So the final answer is: December 6, 1952
Question: Who was the maternal grandfather of George Washington?
Are follow up questions needed here: Yes.
Follow up: Who was the mother of George Washington?


Follow up: Who was the mother of George Washington?
Intermediate answer: The mother of George Washington was Mary Ball Washington.
Follow up: Who was the father of Mary Ball Washington?
Intermediate answer: The father of Mary Ball Washington was Joseph Ball.
So the final answer is: Joseph Ball
Question: Are both the directors of Jaws and Casino Royale from the same country?
Are follow up questions needed here: Yes.
Follow up: Who is the director of Jaws?
Intermediate Answer: The director of Jaws is Steven Spielberg.
Follow up: Where is Steven Spielberg from?
Intermediate Answer: The United States.
Follow up: Who is the director of Casino Royale?
Intermediate Answer: The director of Casino Royale is Martin Campbell.
Follow up: Where is Martin Campbell from?
Intermediate Answer: New Zealand.
So the final answer is: No
Question: Who was the father of Mary Ball Washington?
Using an example selector#
Feed examples into ExampleSelector#
We will reuse the example set and the formatter from the previous section. However, instead of feeding the examples directly into the FewShotPromptTemplate object, we will feed them into an ExampleSelector object.
In this tutorial, we will use the SemanticSimilarityExampleSelector class. This class selects few shot examples based on their similarity to the input. It uses an embedding model to compute the similarity between the input and the few shot examples, as well as a vector store to perform the nearest neighbor search.
from langchain.prompts.example_selector import SemanticSimilarityExampleSelector
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
example_selector = SemanticSimilarityExampleSelector.from_examples(
    # This is the list of examples available to select from.
    examples,
    # This is the embedding class used to produce embeddings which are used to measure semantic similarity.
    OpenAIEmbeddings(),


OpenAIEmbeddings(),
    # This is the VectorStore class that is used to store the embeddings and do a similarity search over.
    Chroma,
    # This is the number of examples to produce.
    k=1
)
# Select the most similar example to the input.
question = "Who was the father of Mary Ball Washington?"
selected_examples = example_selector.select_examples({"question": question})
print(f"Examples most similar to the input: {question}")
for example in selected_examples:
    print("\n")
    for k, v in example.items():
        print(f"{k}: {v}")
Running Chroma using direct local API.
Using DuckDB in-memory for database. Data will be transient.
Examples most similar to the input: Who was the father of Mary Ball Washington?
question: Who was the maternal grandfather of George Washington?
answer:
Are follow up questions needed here: Yes.
Follow up: Who was the mother of George Washington?
Intermediate answer: The mother of George Washington was Mary Ball Washington.
Follow up: Who was the father of Mary Ball Washington?
Intermediate answer: The father of Mary Ball Washington was Joseph Ball.
So the final answer is: Joseph Ball
Feed example selector into FewShotPromptTemplate#
Finally, create a FewShotPromptTemplate object. This object takes in the example selector and the formatter for the few shot examples.
prompt = FewShotPromptTemplate(
    example_selector=example_selector,
    example_prompt=example_prompt,
    suffix="Question: {input}",
    input_variables=["input"]
)
print(prompt.format(input="Who was the father of Mary Ball Washington?"))
Question: Who was the maternal grandfather of George Washington?
Are follow up questions needed here: Yes.
Follow up: Who was the mother of George Washington?


Follow up: Who was the mother of George Washington?
Intermediate answer: The mother of George Washington was Mary Ball Washington.
Follow up: Who was the father of Mary Ball Washington?
Intermediate answer: The father of Mary Ball Washington was Joseph Ball.
So the final answer is: Joseph Ball
Question: Who was the father of Mary Ball Washington?
previous
Create a custom example selector
next
Prompt Serialization
 Contents

Use Case
Using an example set
Create the example set
Create a formatter for the few shot examples
Feed examples and formatter to FewShotPromptTemplate
Using an example selector
Feed examples into ExampleSelector
Feed example selector into FewShotPromptTemplate
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
Output Parsers
 Contents
Output Parsers
PydanticOutputParser
Fixing Output Parsing Mistakes
Fixing Output Parsing Mistakes with the original prompt
Older, less powerful parsers
Structured Output Parser
CommaSeparatedListOutputParser
Output Parsers#
Language models output text. But many times you may want to get more structured information than just text back. This is where output parsers come in.
Output parsers are classes that help structure language model responses. There are two main methods an output parser must implement:
get_format_instructions() -> str: A method which returns a string containing instructions for how the output of a language model should be formatted.
parse(str) -> Any: A method which takes in a string (assumed to be the response from a language model) and parses it into some structure.
And then one optional one:
parse_with_prompt(str) -> Any: A method which takes in a string (assumed to be the response from a language model) and a prompt (assumed to the prompt that generated such a response) and parses it into some structure. The prompt is largely provided in the event the OutputParser wants to retry or fix the output in some way, and needs information from the prompt to do so.
Below we go over some examples of output parsers.
from langchain.prompts import PromptTemplate, ChatPromptTemplate, HumanMessagePromptTemplate
from langchain.llms import OpenAI
from langchain.chat_models import ChatOpenAI
PydanticOutputParser#
This output parser allows users to specify an arbitrary JSON schema and query LLMs for JSON outputs that conform to that schema.
Keep in mind that large language models are leaky abstractions! You’ll have to use an LLM with sufficient capacity to generate well-formed JSON. In the OpenAI family, DaVinci can do reliably but Curie’s ability already drops off dramatically.


Use Pydantic to declare your data model. Pydantic’s BaseModel like a Python dataclass, but with actual type checking + coercion.
from langchain.output_parsers import PydanticOutputParser
from pydantic import BaseModel, Field, validator
from typing import List
model_name = 'text-davinci-003'
temperature = 0.0
model = OpenAI(model_name=model_name, temperature=temperature)
# Define your desired data structure.
class Joke(BaseModel):
    setup: str = Field(description="question to set up a joke")
    punchline: str = Field(description="answer to resolve the joke")

    # You can add custom validation logic easily with Pydantic.
    @validator('setup')
    def question_ends_with_question_mark(cls, field):
        if field[-1] != '?':
            raise ValueError("Badly formed question!")
        return field
# And a query intented to prompt a language model to populate the data structure.
joke_query = "Tell me a joke."
# Set up a parser + inject instructions into the prompt template.
parser = PydanticOutputParser(pydantic_object=Joke)
prompt = PromptTemplate(
    template="Answer the user query.\n{format_instructions}\n{query}\n",
    input_variables=["query"],
    partial_variables={"format_instructions": parser.get_format_instructions()}
)
_input = prompt.format_prompt(query=joke_query)
output = model(_input.to_string())
parser.parse(output)
Joke(setup='Why did the chicken cross the road?', punchline='To get to the other side!')
# Here's another example, but with a compound typed field.
class Actor(BaseModel):
    name: str = Field(description="name of an actor")


class Actor(BaseModel):
    name: str = Field(description="name of an actor")
    film_names: List[str] = Field(description="list of names of films they starred in")

actor_query = "Generate the filmography for a random actor."
parser = PydanticOutputParser(pydantic_object=Actor)
prompt = PromptTemplate(
    template="Answer the user query.\n{format_instructions}\n{query}\n",
    input_variables=["query"],
    partial_variables={"format_instructions": parser.get_format_instructions()}
)
_input = prompt.format_prompt(query=actor_query)
output = model(_input.to_string())
parser.parse(output)
Actor(name='Tom Hanks', film_names=['Forrest Gump', 'Saving Private Ryan', 'The Green Mile', 'Cast Away', 'Toy Story'])
Fixing Output Parsing Mistakes#
The above guardrail simply tries to parse the LLM response. If it does not parse correctly, then it errors.
But we can do other things besides throw errors. Specifically, we can pass the misformatted output, along with the formatted instructions, to the model and ask it to fix it.
For this example, we’ll use the above OutputParser. Here’s what happens if we pass it a result that does not comply with the schema:
misformatted = "{'name': 'Tom Hanks', 'film_names': ['Forrest Gump']}"
parser.parse(misformatted)
---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
File ~/workplace/langchain/langchain/output_parsers/pydantic.py:23, in PydanticOutputParser.parse(self, text)
     22     json_str = match.group()
---> 23 json_object = json.loads(json_str)
     24 return self.pydantic_object.parse_obj(json_object)


24 return self.pydantic_object.parse_obj(json_object)
File ~/.pyenv/versions/3.9.1/lib/python3.9/json/__init__.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    343 if (cls is None and object_hook is None and
    344         parse_int is None and parse_float is None and
    345         parse_constant is None and object_pairs_hook is None and not kw):
--> 346     return _default_decoder.decode(s)
    347 if cls is None:
File ~/.pyenv/versions/3.9.1/lib/python3.9/json/decoder.py:337, in JSONDecoder.decode(self, s, _w)
    333 """Return the Python representation of ``s`` (a ``str`` instance
    334 containing a JSON document).
    335
    336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338 end = _w(s, end).end()
File ~/.pyenv/versions/3.9.1/lib/python3.9/json/decoder.py:353, in JSONDecoder.raw_decode(self, s, idx)
    352 try:
--> 353     obj, end = self.scan_once(s, idx)
    354 except StopIteration as err:
JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
During handling of the above exception, another exception occurred:
OutputParserException                     Traceback (most recent call last)
Cell In[7], line 1
----> 1 parser.parse(misformatted)


Cell In[7], line 1
----> 1 parser.parse(misformatted)
File ~/workplace/langchain/langchain/output_parsers/pydantic.py:29, in PydanticOutputParser.parse(self, text)
     27 name = self.pydantic_object.__name__
     28 msg = f"Failed to parse {name} from completion {text}. Got: {e}"
---> 29 raise OutputParserException(msg)
OutputParserException: Failed to parse Actor from completion {'name': 'Tom Hanks', 'film_names': ['Forrest Gump']}. Got: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
Now we can construct and use a OutputFixingParser. This output parser takes as an argument another output parser but also an LLM with which to try to correct any formatting mistakes.
from langchain.output_parsers import OutputFixingParser
new_parser = OutputFixingParser.from_llm(parser=parser, llm=ChatOpenAI())
new_parser.parse(misformatted)
Actor(name='Tom Hanks', film_names=['Forrest Gump'])
Fixing Output Parsing Mistakes with the original prompt#
While in some cases it is possible to fix any parsing mistakes by only looking at the output, in other cases it can’t. An example of this is when the output is not just in the incorrect format, but is partially complete. Consider the below example.
template = """Based on the user question, provide an Action and Action Input for what step should be taken.
{format_instructions}
Question: {query}
Response:"""
class Action(BaseModel):
    action: str = Field(description="action to take")
    action_input: str = Field(description="input to the action")

parser = PydanticOutputParser(pydantic_object=Action)


parser = PydanticOutputParser(pydantic_object=Action)
prompt = PromptTemplate(
    template="Answer the user query.\n{format_instructions}\n{query}\n",
    input_variables=["query"],
    partial_variables={"format_instructions": parser.get_format_instructions()}
)
prompt_value = prompt.format_prompt(query="who is leo di caprios gf?")
bad_response = '{"action": "search"}'
If we try to parse this response as is, we will get an error
parser.parse(bad_response)
---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
File ~/workplace/langchain/langchain/output_parsers/pydantic.py:24, in PydanticOutputParser.parse(self, text)
     23     json_object = json.loads(json_str)
---> 24     return self.pydantic_object.parse_obj(json_object)
     26 except (json.JSONDecodeError, ValidationError) as e:
File ~/.pyenv/versions/3.9.1/envs/langchain/lib/python3.9/site-packages/pydantic/main.py:527, in pydantic.main.BaseModel.parse_obj()
File ~/.pyenv/versions/3.9.1/envs/langchain/lib/python3.9/site-packages/pydantic/main.py:342, in pydantic.main.BaseModel.__init__()
ValidationError: 1 validation error for Action
action_input
  field required (type=value_error.missing)
During handling of the above exception, another exception occurred:
OutputParserException                     Traceback (most recent call last)
Cell In[15], line 1
----> 1 parser.parse(bad_response)
File ~/workplace/langchain/langchain/output_parsers/pydantic.py:29, in PydanticOutputParser.parse(self, text)


27 name = self.pydantic_object.__name__
     28 msg = f"Failed to parse {name} from completion {text}. Got: {e}"
---> 29 raise OutputParserException(msg)
OutputParserException: Failed to parse Action from completion {"action": "search"}. Got: 1 validation error for Action
action_input
  field required (type=value_error.missing)
If we try to use the OutputFixingParser to fix this error, it will be confused - namely, it doesn’t know what to actually put for action input.
fix_parser = OutputFixingParser.from_llm(parser=parser, llm=ChatOpenAI())
fix_parser.parse(bad_response)
Action(action='search', action_input='keyword')
Instead, we can use the RetryOutputParser, which passes in the prompt (as well as the original output) to try again to get a better response.
from langchain.output_parsers import RetryWithErrorOutputParser
retry_parser = RetryWithErrorOutputParser.from_llm(parser=parser, llm=ChatOpenAI())
retry_parser.parse_with_prompt(bad_response, prompt_value)
Action(action='search', action_input='leo di caprios girlfriend')
Older, less powerful parsers#
Structured Output Parser#
While the Pydantic/JSON parser is more powerful, we initially experimented data structures having text fields only.
from langchain.output_parsers import StructuredOutputParser, ResponseSchema
Here we define the response schema we want to receive.
response_schemas = [
    ResponseSchema(name="answer", description="answer to the user's question"),
    ResponseSchema(name="source", description="source used to answer the user's question, should be a website.")
]
output_parser = StructuredOutputParser.from_response_schemas(response_schemas)


]
output_parser = StructuredOutputParser.from_response_schemas(response_schemas)
We now get a string that contains instructions for how the response should be formatted, and we then insert that into our prompt.
format_instructions = output_parser.get_format_instructions()
prompt = PromptTemplate(
    template="answer the users question as best as possible.\n{format_instructions}\n{question}",
    input_variables=["question"],
    partial_variables={"format_instructions": format_instructions}
)
We can now use this to format a prompt to send to the language model, and then parse the returned result.
model = OpenAI(temperature=0)
_input = prompt.format_prompt(question="what's the capital of france")
output = model(_input.to_string())
output_parser.parse(output)
{'answer': 'Paris', 'source': 'https://en.wikipedia.org/wiki/Paris'}
And here’s an example of using this in a chat model
chat_model = ChatOpenAI(temperature=0)
prompt = ChatPromptTemplate(
    messages=[
        HumanMessagePromptTemplate.from_template("answer the users question as best as possible.\n{format_instructions}\n{question}")
    ],
    input_variables=["question"],
    partial_variables={"format_instructions": format_instructions}
)
_input = prompt.format_prompt(question="what's the capital of france")
output = chat_model(_input.to_messages())
output_parser.parse(output.content)
{'answer': 'Paris', 'source': 'https://en.wikipedia.org/wiki/Paris'}
CommaSeparatedListOutputParser#
Here’s another parser strictly less powerful than Pydantic/JSON parsing.
from langchain.output_parsers import CommaSeparatedListOutputParser
output_parser = CommaSeparatedListOutputParser()
format_instructions = output_parser.get_format_instructions()
prompt = PromptTemplate(


format_instructions = output_parser.get_format_instructions()
prompt = PromptTemplate(
    template="List five {subject}.\n{format_instructions}",
    input_variables=["subject"],
    partial_variables={"format_instructions": format_instructions}
)
model = OpenAI(temperature=0)
_input = prompt.format(subject="ice cream flavors")
output = model(_input)
output_parser.parse(output)
['Vanilla',
 'Chocolate',
 'Strawberry',
 'Mint Chocolate Chip',
 'Cookies and Cream']
previous
Example Selectors
next
Prompts
 Contents

Output Parsers
PydanticOutputParser
Fixing Output Parsing Mistakes
Fixing Output Parsing Mistakes with the original prompt
Older, less powerful parsers
Structured Output Parser
CommaSeparatedListOutputParser
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.md
.pdf
Create a custom example selector
 Contents
Implement custom example selector
Use custom example selector
Create a custom example selector#
In this tutorial, we’ll create a custom example selector that selects every alternate example from a given list of examples.
An ExampleSelector must implement two methods:
An add_example method which takes in an example and adds it into the ExampleSelector
A select_examples method which takes in input variables (which are meant to be user input) and returns a list of examples to use in the few shot prompt.
Let’s implement a custom ExampleSelector that just selects two examples at random.
Note
Take a look at the current set of example selector implementations supported in LangChain here.
Implement custom example selector#
from langchain.prompts.example_selector.base import BaseExampleSelector
from typing import Dict, List
import numpy as np
class CustomExampleSelector(BaseExampleSelector):

    def __init__(self, examples: List[Dict[str, str]]):
        self.examples = examples

    def add_example(self, example: Dict[str, str]) -> None:
        """Add new example to store for a key."""
        self.examples.append(example)
    def select_examples(self, input_variables: Dict[str, str]) -> List[dict]:
        """Select which examples to use based on the inputs."""
        return np.random.choice(self.examples, size=2, replace=False)
Use custom example selector#
examples = [
    {"foo": "1"},
    {"foo": "2"},
    {"foo": "3"}
]
# Initialize example selector.
example_selector = CustomExampleSelector(examples)
# Select examples
example_selector.select_examples({"foo": "foo"})
# -> array([{'foo': '2'}, {'foo': '3'}], dtype=object)
# Add new example to the set of examples


# Add new example to the set of examples
example_selector.add_example({"foo": "4"})
example_selector.examples
# -> [{'foo': '1'}, {'foo': '2'}, {'foo': '3'}, {'foo': '4'}]
# Select examples
example_selector.select_examples({"foo": "foo"})
# -> array([{'foo': '1'}, {'foo': '4'}], dtype=object)
previous
Create a custom prompt template
next
Provide few shot examples to a prompt
 Contents

Implement custom example selector
Use custom example selector
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
Create a custom prompt template
 Contents
Why are custom prompt templates needed?
Creating a Custom Prompt Template
Use the custom prompt template
Create a custom prompt template#
Let’s suppose we want the LLM to generate English language explanations of a function given its name. To achieve this task, we will create a custom prompt template that takes in the function name as input, and formats the prompt template to provide the source code of the function.
Why are custom prompt templates needed?#
LangChain provides a set of default prompt templates that can be used to generate prompts for a variety of tasks. However, there may be cases where the default prompt templates do not meet your needs. For example, you may want to create a prompt template with specific dynamic instructions for your language model. In such cases, you can create a custom prompt template.
Take a look at the current set of default prompt templates here.
Creating a Custom Prompt Template#
There are essentially two distinct prompt templates available - string prompt templates and chat prompt templates. String prompt templates provides a simple prompt in string format, while chat prompt templates produces a more structured prompt to be used with a chat API.
In this guide, we will create a custom prompt using a string prompt template.
To create a custom string prompt template, there are two requirements:
It has an input_variables attribute that exposes what input variables the prompt template expects.
It exposes a format method that takes in keyword arguments corresponding to the expected input_variables and returns the formatted prompt.
We will create a custom prompt template that takes in the function name as input and formats the prompt to provide the source code of the function. To achieve this, let’s first create a function that will return the source code of a function given its name.
import inspect
def get_source_code(function_name):
    # Get the source code of the function
    return inspect.getsource(function_name)


# Get the source code of the function
    return inspect.getsource(function_name)
Next, we’ll create a custom prompt template that takes in the function name as input, and formats the prompt template to provide the source code of the function.
from langchain.prompts import StringPromptTemplate
from pydantic import BaseModel, validator
class FunctionExplainerPromptTemplate(StringPromptTemplate, BaseModel):
    """ A custom prompt template that takes in the function name as input, and formats the prompt template to provide the source code of the function. """
    @validator("input_variables")
    def validate_input_variables(cls, v):
        """ Validate that the input variables are correct. """
        if len(v) != 1 or "function_name" not in v:
            raise ValueError("function_name must be the only input_variable.")
        return v
    def format(self, **kwargs) -> str:
        # Get the source code of the function
        source_code = get_source_code(kwargs["function_name"])
        # Generate the prompt to be sent to the language model
        prompt = f"""
        Given the function name and source code, generate an English language explanation of the function.
        Function Name: {kwargs["function_name"].__name__}
        Source Code:
        {source_code}
        Explanation:
        """
        return prompt

    def _prompt_type(self):
        return "function-explainer"
Use the custom prompt template#
Now that we have created a custom prompt template, we can use it to generate prompts for our task.
fn_explainer = FunctionExplainerPromptTemplate(input_variables=["function_name"])
# Generate a prompt for the function "get_source_code"
prompt = fn_explainer.format(function_name=get_source_code)
print(prompt)


prompt = fn_explainer.format(function_name=get_source_code)
print(prompt)
        Given the function name and source code, generate an English language explanation of the function.
        Function Name: get_source_code
        Source Code:
        def get_source_code(function_name):
    # Get the source code of the function
    return inspect.getsource(function_name)
        Explanation:

previous
How-To Guides
next
Create a custom example selector
 Contents

Why are custom prompt templates needed?
Creating a Custom Prompt Template
Use the custom prompt template
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.md
.pdf
Key Concepts
 Contents
Document
Loader
Unstructured
Key Concepts#
Document#
This class is a container for document information. This contains two parts:
page_content: The content of the actual page itself.
metadata: The metadata associated with the document. This can be things like the file path, the url, etc.
Loader#
This base class is a way to load documents. It exposes a load method that returns Document objects.
Unstructured#
Unstructured is a python package specifically focused on transformations from raw documents to text.
previous
Document Loaders
next
How To Guides
 Contents

Document
Loader
Unstructured
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.rst
.pdf
How To Guides
How To Guides#
There are a lot of different document loaders that LangChain supports. Below are how-to guides for working with them
File Loader: A walkthrough of how to use Unstructured to load files of arbitrary types (pdfs, txt, html, etc).
Directory Loader: A walkthrough of how to use Unstructured load files from a given directory.
Notion: A walkthrough of how to load data for an arbitrary Notion DB.
ReadTheDocs: A walkthrough of how to load data for documentation generated by ReadTheDocs.
HTML: A walkthrough of how to load data from an html file.
PDF: A walkthrough of how to load data from a PDF file.
PowerPoint: A walkthrough of how to load data from a powerpoint file.
Email: A walkthrough of how to load data from an email (.eml) file.
GoogleDrive: A walkthrough of how to load data from Google drive.
Microsoft Word: A walkthrough of how to load data from Microsoft Word files.
Obsidian: A walkthrough of how to load data from an Obsidian file dump.
Roam: A walkthrough of how to load data from a Roam file export.
EverNote: A walkthrough of how to load data from a EverNote (.enex) file.
YouTube: A walkthrough of how to load the transcript from a YouTube video.
Hacker News: A walkthrough of how to load a Hacker News page.
GitBook: A walkthrough of how to load a GitBook page.
s3 File: A walkthrough of how to load a file from s3.
s3 Directory: A walkthrough of how to load all files in a directory from s3.
GCS File: A walkthrough of how to load a file from Google Cloud Storage (GCS).


GCS File: A walkthrough of how to load a file from Google Cloud Storage (GCS).
GCS Directory: A walkthrough of how to load all files in a directory from Google Cloud Storage (GCS).
Web Base: A walkthrough of how to load all text data from webpages.
IMSDb: A walkthrough of how to load all text data from IMSDb webpage.
AZLyrics: A walkthrough of how to load all text data from AZLyrics webpage.
College Confidential: A walkthrough of how to load all text data from College Confidential webpage.
Gutenberg: A walkthrough of how to load data from a Gutenberg ebook text.
Airbyte Json: A walkthrough of how to load data from a local Airbyte JSON file.
CoNLL-U: A walkthrough of how to load data from a ConLL-U file.
iFixit: A walkthrough of how to search and load data like guides, technical Q&A’s, and device wikis from iFixit.com
Blackboard: A walkthrough of how to load data from a Blackboard course.
previous
Key Concepts
next
CoNLL-U
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
Airbyte JSON
Airbyte JSON#
This covers how to load any source from Airbyte into a local JSON file that can be read in as a document
Prereqs:
Have docker desktop installed
Steps:
Clone Airbyte from GitHub - git clone https://github.com/airbytehq/airbyte.git
Switch into Airbyte directory - cd airbyte
Start Airbyte - docker compose up
In your browser, just visit http://localhost:8000. You will be asked for a username and password. By default, that’s username airbyte and password password.
Setup any source you wish.
Set destination as Local JSON, with specified destination path - lets say /json_data. Set up manual sync.
Run the connection!
To see what files are create, you can navigate to: file:///tmp/airbyte_local
Find your data and copy path. That path should be saved in the file variable below. It should start with /tmp/airbyte_local
from langchain.document_loaders import AirbyteJSONLoader
!ls /tmp/airbyte_local/json_data/
_airbyte_raw_pokemon.jsonl
loader = AirbyteJSONLoader('/tmp/airbyte_local/json_data/_airbyte_raw_pokemon.jsonl')
data = loader.load()
print(data[0].page_content[:500])
abilities:
ability:
name: blaze
url: https://pokeapi.co/api/v2/ability/66/
is_hidden: False
slot: 1
ability:
name: solar-power
url: https://pokeapi.co/api/v2/ability/94/
is_hidden: True
slot: 3
base_experience: 267
forms:
name: charizard
url: https://pokeapi.co/api/v2/pokemon-form/6/
game_indices:


game_indices:
game_index: 180
version:
name: red
url: https://pokeapi.co/api/v2/version/1/
game_index: 180
version:
name: blue
url: https://pokeapi.co/api/v2/version/2/
game_index: 180
version:
n
previous
CoNLL-U
next
AZLyrics
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
PowerPoint
 Contents
Retain Elements
PowerPoint#
This covers how to load PowerPoint documents into a document format that we can use downstream.
from langchain.document_loaders import UnstructuredPowerPointLoader
loader = UnstructuredPowerPointLoader("example_data/fake-power-point.pptx")
data = loader.load()
data
[Document(page_content='Adding a Bullet Slide\n\nFind the bullet slide layout\n\nUse _TextFrame.text for first bullet\n\nUse _TextFrame.add_paragraph() for subsequent bullets\n\nHere is a lot of text!\n\nHere is some text in a text box!', lookup_str='', metadata={'source': 'example_data/fake-power-point.pptx'}, lookup_index=0)]
Retain Elements#
Under the hood, Unstructured creates different “elements” for different chunks of text. By default we combine those together, but you can easily keep that separation by specifying mode="elements".
loader = UnstructuredPowerPointLoader("example_data/fake-power-point.pptx", mode="elements")
data = loader.load()
data[0]
Document(page_content='Adding a Bullet Slide', lookup_str='', metadata={'source': 'example_data/fake-power-point.pptx'}, lookup_index=0)
previous
PDF
next
ReadTheDocs Documentation
 Contents

Retain Elements
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
Images
 Contents
Using Unstructured
Retain Elements
Images#
This covers how to load images such as JPGs PNGs into a document format that we can use downstream.
Using Unstructured#
from langchain.document_loaders.image import UnstructuredImageLoader
loader = UnstructuredImageLoader("layout-parser-paper-fast.jpg")
data = loader.load()
data[0]


Document(page_content="LayoutParser: A Unified Toolkit for Deep\nLearning Based Document Image Analysis\n\n\n‘Zxjiang Shen' (F3}, Ruochen Zhang”, Melissa Dell*, Benjamin Charles Germain\nLeet, Jacob Carlson, and Weining LiF\n\n\nsugehen\n\nshangthrows, et\n\n“Abstract. Recent advanocs in document image analysis (DIA) have been\n‘pimarliy driven bythe application of neural networks dell roar\n{uteomer could be aly deployed in production and extended fo farther\n[nvetigtion. However, various factory ke lcely organize codebanee\nsnd sophisticated modal cnigurations compat the ey ree of\n‘erin! innovation by wide sence, Though there have been sng\n‘Hors to improve reuablty and simplify deep lees (DL) mode\n‘aon, sone of them ae optimized for challenge inthe demain of DIA,\nThis roprscte a major gap in the extng fol, sw DIA i eal to\nscademic research acon wie range of dpi in the social ssencee\n[rary for streamlining the sage of DL in DIA research and


streamlining the sage of DL in DIA research and appicn\n‘tons The core LayoutFaraer brary comes with a sch of simple and\nIntative interfaee or applying and eutomiing DI. odel fr Inyo de\npltfom for sharing both protrined modes an fal document dist\n{ation pipeline We demonutate that LayootPareer shea fr both\nlightweight and lrgeseledgtieation pipelines in eal-word uae ces\nThe leary pblely smal at Btspe://layost-pareergsthab So\n\n\n\n‘Keywords: Document Image Analysis» Deep Learning Layout Analysis\n‘Character Renguition - Open Serres dary « Tol\n\n\nIntroduction\n\n\n‘Deep Learning(DL)-based approaches are the state-of-the-art for a wide range of\ndoctiment image analysis (DIA) tea including document image clasiffeation [I]\n", lookup_str='', metadata={'source': 'layout-parser-paper-fast.jpg'}, lookup_index=0)


Retain Elements#
Under the hood, Unstructured creates different “elements” for different chunks of text. By default we combine those together, but you can easily keep that separation by specifying mode="elements".
loader = UnstructuredImageLoader("layout-parser-paper-fast.jpg", mode="elements")
data = loader.load()
data[0]
Document(page_content='LayoutParser: A Unified Toolkit for Deep\nLearning Based Document Image Analysis\n', lookup_str='', metadata={'source': 'layout-parser-paper-fast.jpg', 'filename': 'layout-parser-paper-fast.jpg', 'page_number': 1, 'category': 'Title'}, lookup_index=0)
previous
iFixit
next
IMSDb
 Contents

Using Unstructured
Retain Elements
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
Subtitle Files
Subtitle Files#
How to load data from subtitle (.srt) files
from langchain.document_loaders import SRTLoader
loader = SRTLoader("example_data/Star_Wars_The_Clone_Wars_S06E07_Crisis_at_the_Heart.srt")
docs = loader.load()
docs[0].page_content[:100]
'<i>Corruption discovered\nat the core of the Banking Clan!</i> <i>Reunited, Rush Clovis\nand Senator A'
previous
s3 File
next
Telegram
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
Notion
 Contents
🧑 Instructions for ingesting your own dataset
Notion#
This notebook covers how to load documents from a Notion database dump.
In order to get this notion dump, follow these instructions:
🧑 Instructions for ingesting your own dataset#
Export your dataset from Notion. You can do this by clicking on the three dots in the upper right hand corner and then clicking Export.
When exporting, make sure to select the Markdown & CSV format option.
This will produce a .zip file in your Downloads folder. Move the .zip file into this repository.
Run the following command to unzip the zip file (replace the Export... with your own file name as needed).
unzip Export-d3adfe0f-3131-4bf3-8987-a52017fc1bae.zip -d Notion_DB
Run the following command to ingest the data.
from langchain.document_loaders import NotionDirectoryLoader
loader = NotionDirectoryLoader("Notion_DB")
docs = loader.load()
previous
Notebook
next
Obsidian
 Contents

🧑 Instructions for ingesting your own dataset
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
EverNote
EverNote#
How to load EverNote file from disk.
# !pip install pypandoc
# import pypandoc
# pypandoc.download_pandoc()
from langchain.document_loaders import EverNoteLoader
loader = EverNoteLoader("example_data/testing.enex")
loader.load()
[Document(page_content='testing this\n\nwhat happens?\n\nto the world?\n', lookup_str='', metadata={'source': 'example_data/testing.enex'}, lookup_index=0)]
previous
Email
next
Facebook Chat
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
GCS Directory
 Contents
Specifying a prefix
GCS Directory#
This covers how to load document objects from an Google Cloud Storage (GCS) directory.
from langchain.document_loaders import GCSDirectoryLoader
# !pip install google-cloud-storage
loader = GCSDirectoryLoader(project_name="aist", bucket="testing-hwc")
loader.load()
/Users/harrisonchase/workplace/langchain/.venv/lib/python3.10/site-packages/google/auth/_default.py:83: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a "quota exceeded" or "API not enabled" error. We recommend you rerun `gcloud auth application-default login` and make sure a quota project is added. Or you can use service accounts instead. For more information about service accounts, see https://cloud.google.com/docs/authentication/
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
/Users/harrisonchase/workplace/langchain/.venv/lib/python3.10/site-packages/google/auth/_default.py:83: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a "quota exceeded" or "API not enabled" error. We recommend you rerun `gcloud auth application-default login` and make sure a quota project is added. Or you can use service accounts instead. For more information about service accounts, see https://cloud.google.com/docs/authentication/
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmpz37njh7u/fake.docx'}, lookup_index=0)]
Specifying a prefix#


Specifying a prefix#
You can also specify a prefix for more finegrained control over what files to load.
loader = GCSDirectoryLoader(project_name="aist", bucket="testing-hwc", prefix="fake")
loader.load()
/Users/harrisonchase/workplace/langchain/.venv/lib/python3.10/site-packages/google/auth/_default.py:83: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a "quota exceeded" or "API not enabled" error. We recommend you rerun `gcloud auth application-default login` and make sure a quota project is added. Or you can use service accounts instead. For more information about service accounts, see https://cloud.google.com/docs/authentication/
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
/Users/harrisonchase/workplace/langchain/.venv/lib/python3.10/site-packages/google/auth/_default.py:83: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a "quota exceeded" or "API not enabled" error. We recommend you rerun `gcloud auth application-default login` and make sure a quota project is added. Or you can use service accounts instead. For more information about service accounts, see https://cloud.google.com/docs/authentication/
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmpylg6291i/fake.docx'}, lookup_index=0)]
previous
Facebook Chat
next
GCS File Storage
 Contents

Specifying a prefix
By Harrison Chase

      © Copyright 2023, Harrison Chase.


By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
HTML
HTML#
This covers how to load HTML documents into a document format that we can use downstream.
from langchain.document_loaders import UnstructuredHTMLLoader
loader = UnstructuredHTMLLoader("example_data/fake-content.html")
data = loader.load()
data
[Document(page_content='My First Heading\n\nMy first paragraph.', lookup_str='', metadata={'source': 'example_data/fake-content.html'}, lookup_index=0)]
previous
Hacker News
next
iFixit
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
Hacker News
Hacker News#
How to pull page data and comments from Hacker News
from langchain.document_loaders import HNLoader
loader = HNLoader("https://news.ycombinator.com/item?id=34817881")
data = loader.load()
data
[Document(page_content="delta_p_delta_x 18 hours ago  \n             | next [–] \n\nAstrophysical and cosmological simulations are often insightful. They're also very cross-disciplinary; besides the obvious astrophysics, there's networking and sysadmin, parallel computing and algorithm theory (so that the simulation programs are actually fast but still accurate), systems design, and even a bit of graphic design for the visualisations.Some of my favourite simulation projects:- IllustrisTNG: https://www.tng-project.org/- SWIFT: https://swift.dur.ac.uk/- CO5BOLD: https://www.astro.uu.se/~bf/co5bold_main.html (which produced these animations of a red-giant star: https://www.astro.uu.se/~bf/movie/AGBmovie.html)- AbacusSummit: https://abacussummit.readthedocs.io/en/latest/And I can add the simulations in the article, too.\n \nreply", lookup_str='', metadata={'source': 'https://news.ycombinator.com/item?id=34817881', 'title': 'What Lights the Universe’s Standard Candles?'}, lookup_index=0),


Document(page_content="andrewflnr 19 hours ago  \n             | prev | next [–] \n\nWhoa. I didn't know the accretion theory of Ia supernovae was dead, much less that it had been since 2011.\n \nreply", lookup_str='', metadata={'source': 'https://news.ycombinator.com/item?id=34817881', 'title': 'What Lights the Universe’s Standard Candles?'}, lookup_index=0),
 Document(page_content='andreareina 18 hours ago  \n             | prev | next [–] \n\nThis seems  to be the paper https://academic.oup.com/mnras/article/517/4/5260/6779709\n \nreply', lookup_str='', metadata={'source': 'https://news.ycombinator.com/item?id=34817881', 'title': 'What Lights the Universe’s Standard Candles?'}, lookup_index=0),
 Document(page_content="andreareina 18 hours ago  \n             | prev [–] \n\nWouldn't double detonation show up as variance in the brightness?\n \nreply", lookup_str='', metadata={'source': 'https://news.ycombinator.com/item?id=34817881', 'title': 'What Lights the Universe’s Standard Candles?'}, lookup_index=0)]
previous
Gutenberg
next
HTML
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
Gutenberg
Gutenberg#
This covers how to load links to Gutenberg e-books into a document format that we can use downstream.
from langchain.document_loaders import GutenbergLoader
loader = GutenbergLoader('https://www.gutenberg.org/cache/epub/69972/pg69972.txt')
data = loader.load()
data
previous
Google Drive
next
Hacker News
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
Email
 Contents
Retain Elements
Email#
This notebook shows how to load email (.eml) files.
from langchain.document_loaders import UnstructuredEmailLoader
loader = UnstructuredEmailLoader('example_data/fake-email.eml')
data = loader.load()
data
[Document(page_content='This is a test email to use for unit tests.\n\nImportant points:\n\nRoses are red\n\nViolets are blue', lookup_str='', metadata={'source': 'example_data/fake-email.eml'}, lookup_index=0)]
Retain Elements#
Under the hood, Unstructured creates different “elements” for different chunks of text. By default we combine those together, but you can easily keep that separation by specifying mode="elements".
loader = UnstructuredEmailLoader('example_data/fake-email.eml', mode="elements")
data = loader.load()
data[0]
Document(page_content='This is a test email to use for unit tests.', lookup_str='', metadata={'source': 'example_data/fake-email.eml'}, lookup_index=0)
previous
Directory Loader
next
EverNote
 Contents

Retain Elements
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
Word Documents
 Contents
Retain Elements
Word Documents#
This covers how to load Word documents into a document format that we can use downstream.
from langchain.document_loaders import UnstructuredWordDocumentLoader
loader = UnstructuredWordDocumentLoader("fake.docx")
data = loader.load()
data
[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': 'fake.docx'}, lookup_index=0)]
Retain Elements#
Under the hood, Unstructured creates different “elements” for different chunks of text. By default we combine those together, but you can easily keep that separation by specifying mode="elements".
loader = UnstructuredWordDocumentLoader("fake.docx", mode="elements")
data = loader.load()
data[0]
Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': 'fake.docx', 'filename': 'fake.docx', 'category': 'Title'}, lookup_index=0)
previous
Web Base
next
YouTube
 Contents

Retain Elements
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
Web Base
Web Base#
This covers how to load all text from webpages into a document format that we can use downstream. For more custom logic for loading webpages look at some child class examples such as IMSDbLoader, AZLyricsLoader, and CollegeConfidentialLoader
from langchain.document_loaders import WebBaseLoader
loader = WebBaseLoader("https://www.espn.com/")
data = loader.load()
data


[Document(page_content="\n\n\n\n\n\n\n\n\nESPN - Serving Sports Fans. Anytime. Anywhere.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n        Skip to main content\n    \n\n        Skip to navigation\n    \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<\n\n>\n\n\n\n\n\n\n\n\n\nMenuESPN\n\n\nSearch\n\n\n\nscores\n\n\n\nNFLNBANHLNCAAMNCAAWSoccer…MLBNCAAFGolfTennisSports BettingBoxingCaribbean SeriesCFLNCAACricketF1HorseLLWSMMANASCARNBA G LeagueOlympic SportsRacingRN BBRN FBRugbyWNBAWWEX GamesXFLMore ESPNFantasyListenWatchESPN+\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n  \n\nSUBSCRIBE NOW\n\n\n\n\n\nUFC 284: Makhachev vs. Volkanovski (ESPN+ PPV)\n\n\n\n\n\n\n\nMen's College Hoops: Select


College Hoops: Select Games\n\n\n\n\n\n\n\nWomen's College Hoops: Select Games\n\n\n\n\n\n\n\nNHL: Select Games\n\n\n\n\n\n\n\nGerman Cup: Round of 16\n\n\n\n\n\n\n\n30 For 30: Bullies Of Baltimore\n\n\n\n\n\n\n\nMatt Miller's Two-Round NFL Mock Draft\n\n\nQuick Links\n\n\n\n\nSuper Bowl LVII\n\n\n\n\n\n\n\nSuper Bowl Betting\n\n\n\n\n\n\n\nNBA Trade Machine\n\n\n\n\n\n\n\nNBA All-Star Game\n\n\n\n\n\n\n\nFantasy Baseball: Sign Up\n\n\n\n\n\n\n\nHow To Watch NHL Games\n\n\n\n\n\n\n\nGames For Me\n\n\n\n\n\n\nFavorites\n\n\n\n\n\n\n      Manage Favorites\n      \n\n\n\nCustomize ESPNSign UpLog InESPN Sites\n\n\n\n\nESPN Deportes\n\n\n\n\n\n\n\nAndscape\n\n\n\n\n\n\n\nespnW\n\n\n\n\n\n\n\nESPNFC\n\n\n\n\n\n\n\nX Games\n\n\n\n\n\n\n\nSEC Network\n\n\nESPN Apps\n\n\n\n\nESPN\n\n\n\n\n\n\n\nESPN Fantasy\n\n\nFollow


Fantasy\n\n\nFollow ESPN\n\n\n\n\nFacebook\n\n\n\n\n\n\n\nTwitter\n\n\n\n\n\n\n\nInstagram\n\n\n\n\n\n\n\nSnapchat\n\n\n\n\n\n\n\nYouTube\n\n\n\n\n\n\n\nThe ESPN Daily Podcast\n\n\nAP Photo/Mark J. Terrilllive\n\n\n\nChristian Wood elevates for the big-time stuffChristian Wood elevates for the big-time stuff15m0:29\n\n\nKyrie Irving nails the treyKyrie Irving nails the trey37m0:17\n\n\nDwight Powell rises up for putback dunkDwight Powell throws down the putback dunk for the Mavericks.38m0:16\n\n\nKyrie sinks his first basket with the MavericksKyrie Irving drains the jump shot early vs. the Clippers for his first points with the Mavericks.39m0:17\n\n\nReggie Bullock pulls up for wide open 3Reggie Bullock is left wide open for the 3-pointer early vs. the Clippers.46m0:21\n\n\n\nTOP HEADLINESSources: Lakers get PG Russell in 3-team tradeTrail Blazers shipping Hart to Knicks, sources sayUConn loses two straight for first time


sources sayUConn loses two straight for first time in 30 yearsNFL's Goodell on officiating: Never been betterNFLPA's Smith: Get rid of 'intrusive' NFL combineAlex Morgan: 'Bizarre' for Saudis to sponsor WWCBills' Hamlin makes appearance to receive awardWWE Hall of Famer Lawler recovering from strokeWhich NFL team trades up to No. 1?NBA TRADE DEADLINE3 P.M. ET ON THURSDAYTrade grades: What to make of the three-team deal involving Russell Westbrook and D'Angelo RussellESPN NBA Insider Kevin Pelton is handing out grades for the biggest moves.2hLayne Murdoch Jr./NBAE via Getty ImagesNBA trade tracker: Grades, details for every deal for the 2022-23 seasonWhich players are finding new homes and which teams are making trades during the free-agency frenzy?59mESPN.comNBA trade deadline: Latest buzz and newsNBA SCOREBOARDWEDNESDAY'S GAMESSee AllCLEAR THE RUNWAYJalen Green soars for lefty alley-oop1h0:19Jarrett Allen skies to drop the hammer2h0:16Once


Allen skies to drop the hammer2h0:16Once the undisputed greatest, Joe Montana is still working things out15hWright ThompsonSUPER BOWL LVII6:30 P.M. ET ON SUNDAYBarbershop tales, a fistfight and brotherly love: Untold stories that explain the Kelce brothersJason and Travis Kelce will become the first brothers to face each other in a Super Bowl. Here are untold stories from people who know them best.16hTim McManus, +2 MoreEd Zurga/AP PhotoNFL experts predict Chiefs-Eagles: Our Super Bowl winner picksNFL writers, analysts and reporters take their best guesses on the Super Bowl LVII matchup.17hESPN staffBeware of Philadelphia's Rocky statue curseMadden sim predicts Eagles to win Super BowlTOP 10 TEAMS FALLCOLLEGE HOOPSUConn loses two straight for first time since 1993, falling to Marquette57m1:58Vandy drains 3 at buzzer to knock off Tennessee, fans storm the court1h0:54COLLEGE HOOPS SCORESMEN'S AND WOMEN'S


HOOPS SCORESMEN'S AND WOMEN'S TOP-25 GAMESMen's college hoops scoreboardWomen's college basketball scoresPROJECTING THE BUBBLEMEN'S COLLEGE HOOPSBubble Watch: Current situation? North Carolina has some work to doThe countdown to Selection Sunday on March 12 has begun. We will track which teams are locks and which ones can play their way into or out of the 2023 NCAA men's basketball tournament.6hJohn GasawayAP Photo/Matt Rourke Top HeadlinesSources: Lakers get PG Russell in 3-team tradeTrail Blazers shipping Hart to Knicks, sources sayUConn loses two straight for first time in 30 yearsNFL's Goodell on officiating: Never been betterNFLPA's Smith: Get rid of 'intrusive' NFL combineAlex Morgan: 'Bizarre' for Saudis to sponsor WWCBills' Hamlin makes appearance to receive awardWWE Hall of Famer Lawler recovering from strokeWhich NFL team trades up to No. 1?Favorites FantasyManage FavoritesFantasy HomeCustomize ESPNSign UpLog InICYMI1:54Orlovsky roasts Stephen A. for his top-5 players in the


Stephen A. for his top-5 players in the Super BowlDan Orlovsky lets Stephen A. Smith hear it after he lists his top five players in Super Bowl LVII. Best of ESPN+Michael Hickey/Getty ImagesBubble Watch 2023: Brace yourself for NCAA tournament dramaThe countdown to Selection Sunday on March 12 has begun. We will track which teams are locks and which ones can play their way into or out of the 2023 NCAA men's basketball tournament.Adam Pantozzi/NBAE via Getty ImagesLeBron's journey to the NBA scoring record in shot chartsTake a look at how LeBron James' on-court performance has changed during his march to 38,388 points.Illustration by ESPNRe-drafting first two rounds of 2022 NFL class: All 64 picksWe gave every NFL team a do-over for last year's draft, re-drafting the top 64 picks. Here's who rises and falls with the benefit of hindsight.AP Photo/David DermerWay-too-early 2023 MLB starting rotation rankingsThe Yanks' and Mets' rotations take two of


Yanks' and Mets' rotations take two of the top three spots on our pre-spring training list. Where did they land -- and did another team sneak past one of 'em? Trending NowAP Photo/Jae C. HongStars pay tribute to LeBron James for securing NBA's all-time points recordLeBron James has passed Kareem Abdul-Jabbar for No. 1 on the all-time NBA scoring list, and other stars paid tribute to him on social media.Getty ImagesFans prepare for Rihanna's 2023 Super Bowl halftime showAs Rihanna prepares to make her highly anticipated return, supporters of all 32 teams are paying homage to the icon -- as only tormented NFL fans can.Photo by Cooper Neill/Getty ImagesWhen is the 2023 Super Bowl? Date, time for Chiefs vs. EaglesWe have you covered with seeding, scores and the full schedule for this season's playoffs -- and how to watch Super Bowl LVII.James Drake/Sports Illustrated via Getty ImagesNFL history: Super Bowl winners and resultsFrom the Packers' 1967 win


winners and resultsFrom the Packers' 1967 win over the Chiefs to the Rams' victory over the Bengals in 2022, we've got results for every Super Bowl.China Wong/NHLI via Getty ImagesBoston Bruins record tracker: Wins, points, milestonesThe B's are on pace for NHL records in wins and points, along with some individual superlatives as well. Follow along here with our updated tracker. Sports BettingPhoto by Kevin C. Cox/Getty ImagesSuper Bowl LVII betting: Everything you need to know to bet Eagles-ChiefsHere's your one-stop shop for all the information you need to help make your picks on the Philadelphia Eagles vs. Kansas City Chiefs in Super Bowl LVII. How to Watch on ESPN+(AP Photo/Koji Sasahara, File)How to watch the PGA Tour, Masters, PGA Championship and FedEx Cup playoffs on ESPN, ESPN+Here's everything you need to know about how to watch the PGA Tour, Masters, PGA Championship and FedEx Cup playoffs on ESPN and ESPN+.


and FedEx Cup playoffs on ESPN and ESPN+. \n\nESPN+\n\n\n\n\nUFC 284: Makhachev vs. Volkanovski (ESPN+ PPV)\n\n\n\n\n\n\n\nMen's College Hoops: Select Games\n\n\n\n\n\n\n\nWomen's College Hoops: Select Games\n\n\n\n\n\n\n\nNHL: Select Games\n\n\n\n\n\n\n\nGerman Cup: Round of 16\n\n\n\n\n\n\n\n30 For 30: Bullies Of Baltimore\n\n\n\n\n\n\n\nMatt Miller's Two-Round NFL Mock Draft\n\n\nQuick Links\n\n\n\n\nSuper Bowl LVII\n\n\n\n\n\n\n\nSuper Bowl Betting\n\n\n\n\n\n\n\nNBA Trade Machine\n\n\n\n\n\n\n\nNBA All-Star Game\n\n\n\n\n\n\n\nFantasy Baseball: Sign Up\n\n\n\n\n\n\n\nHow To Watch NHL Games\n\n\n\n\n\n\n\nGames For Me\n\n\nESPN Sites\n\n\n\n\nESPN Deportes\n\n\n\n\n\n\n\nAndscape\n\n\n\n\n\n\n\nespnW\n\n\n\n\n\n\n\nESPNFC\n\n\n\n\n\n\n\nX Games\n\n\n\n\n\n\n\nSEC Network\n\n\nESPN Apps\n\n\n\n\nESPN\n\n\n\n\n\n\n\nESPN Fantasy\n\n\nFollow


Fantasy\n\n\nFollow ESPN\n\n\n\n\nFacebook\n\n\n\n\n\n\n\nTwitter\n\n\n\n\n\n\n\nInstagram\n\n\n\n\n\n\n\nSnapchat\n\n\n\n\n\n\n\nYouTube\n\n\n\n\n\n\n\nThe ESPN Daily Podcast\n\n\nTerms of UsePrivacy PolicyYour US State Privacy RightsChildren's Online Privacy PolicyInterest-Based AdsAbout Nielsen MeasurementDo Not Sell or Share My Personal InformationContact UsDisney Ad Sales SiteWork for ESPNCopyright: © ESPN Enterprises, Inc. All rights reserved.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", lookup_str='', metadata={'source': 'https://www.espn.com/'}, lookup_index=0)]


"""
# Use this piece of code for testing new custom BeautifulSoup parsers
import requests
from bs4 import BeautifulSoup
html_doc = requests.get("{INSERT_NEW_URL_HERE}")
soup = BeautifulSoup(html_doc.text, 'html.parser')
# Beautiful soup logic to be exported to langchain.document_loaders.webpage.py
# Example: transcript = soup.select_one("td[class='scrtext']").text
# BS4 documentation can be found here: https://www.crummy.com/software/BeautifulSoup/bs4/doc/
""";
previous
URL
next
Word Documents
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
Directory Loader
 Contents
Change loader class
Directory Loader#
This covers how to use the DirectoryLoader to load all documents in a directory. Under the hood, by default this uses the UnstructuredLoader
from langchain.document_loaders import DirectoryLoader
We can use the glob parameter to control which files to load. Note that here it doesn’t load the .rst file or the .ipynb files.
loader = DirectoryLoader('../', glob="**/*.md")
docs = loader.load()
len(docs)
1
Change loader class#
By default this uses the UnstructuredLoader class. However, you can change up the type of loader pretty easily.
from langchain.document_loaders import TextLoader
loader = DirectoryLoader('../', glob="**/*.md", loader_cls=TextLoader)
docs = loader.load()
len(docs)
1
previous
CSV Loader
next
Email
 Contents

Change loader class
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
Copy Paste
 Contents
Metadata
Copy Paste#
This notebook covers how to load a document object from something you just want to copy and paste. In this case, you don’t even need to use a DocumentLoader, but rather can just construct the Document directly.
from langchain.docstore.document import Document
text = "..... put the text you copy pasted here......"
doc = Document(page_content=text)
Metadata#
If you want to add metadata about the where you got this piece of text, you easily can with the metadata key.
metadata = {"source": "internet", "date": "Friday"}
doc = Document(page_content=text, metadata=metadata)
previous
College Confidential
next
CSV Loader
 Contents

Metadata
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
URL
URL#
This covers how to load HTML documents from a list of URLs into a document format that we can use downstream.
 from langchain.document_loaders import UnstructuredURLLoader
urls = [
    "https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-8-2023",
    "https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-9-2023"
]
loader = UnstructuredURLLoader(urls=urls)
data = loader.load()
previous
Unstructured File Loader
next
Web Base
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Mar 22, 2023.


.ipynb
.pdf
IMSDb
IMSDb#
This covers how to load IMSDb webpages into a document format that we can use downstream.
from langchain.document_loaders import IMSDbLoader
loader = IMSDbLoader("https://imsdb.com/scripts/BlacKkKlansman.html")
data = loader.load()
data


[Document(page_content='\n\r\n\r\n\r\n\r\n                                    BLACKKKLANSMAN\r\n                         \r\n                         \r\n                         \r\n                         \r\n                                      Written by\r\n\r\n                          Charlie Wachtel & David Rabinowitz\r\n\r\n                                         and\r\n\r\n                              Kevin Willmott & Spike Lee\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n                         FADE IN:\r\n


FADE IN:\r\n                         \r\n          SCENE FROM "GONE WITH THE WIND"\r\n                         \r\n          Scarlett O\'Hara, played by Vivian Leigh, walks through the\r\n          Thousands of injured Confederate Soldiers pulling back to\r\n          reveal the Famous Shot of the tattered Confederate Flag in\r\n          "Gone with the Wind" as The Max Stein Music Score swells from\r\n          Dixie to Taps.\r\n                         \r\n                                   BEAUREGARD- KLAN NARRATOR (O.S.)\r\n                       They say they may have lost the\r\n


Battle but they didn\'t lose The War.\r\n                       Yes, Friends, We are under attack.\r\n                         \r\n                                                                          CUT TO:\r\n                         \r\n          A 1960\'S EDUCATIONAL STYLE FILM\r\n                         \r\n          Shot on Grainy COLOR 16MM EKTACHROME Film, The NARRATOR\r\n          BEAUREGARD, a Middle Aged but handsome, White Male, sits at a\r\n          desk, a Confederate Flag on a stand beside him. Very\r\n          Official. He


him. Very\r\n          Official. He is not a Southerner and speaks with articulation\r\n          and intelligence.\r\n                         \r\n                                   BEAUREGARD- KLAN NARRATOR\r\n                       You\'ve read about it in your Local\r\n                       Newspapers or seen it on The Evening\r\n                       News. That\'s right. We\'re living in\r\n                       an Era marked by the spread of\r\n                       Integration and Miscegenation.\r\n                         \r\n


CUT TO:\r\n                         \r\n          FOOTAGE OF THE LITTLE ROCK NINE\r\n                         \r\n          being escorted into CENTRAL HIGH SCHOOL, Little Rock,\r\n          Arkansas by The National Guard.\r\n                         \r\n                                   BEAUREGARD- KLAN NARRATOR\r\n                                   (V.O.)(CONT\'D)\r\n                       The Brown Decision forced upon us by\r\n                       The Jewish controlled Puppets on the\r\n


Jewish controlled Puppets on the\r\n                       U.S. Supreme Court compelling White\r\n                       children to go to School with an\r\n                       Inferior Race is The Final Nail in a\r\n                       Black Coffin towards America becoming\r\n                       a Mongrel Nation.\r\n                         \r\n          A QUICK SERIES OF IMAGES\r\n                         \r\n          Segregation Signs. Antebellum Photos. Happy Slaves in Old\r\n          Movies. Masters inspecting their Cotton and Tobacco with\r\n          their Slaves in The Fields. Blacks shining Shoes and working\r\n          as


Shoes and working\r\n          as Butlers, Porters and Maids.\r\n                                   BEAUREGARD- KLAN NARRATOR (V.O.)\r\n                                                  (CONT\'D)\r\n                       We had a great way of Life before The\r\n                       Martin Luther Coon\'s of The World...\r\n                         \r\n                                                                          CUT TO:\r\n                         \r\n          The Billboard of Dr. Martin Luther King Jr. sitting in


of Dr. Martin Luther King Jr. sitting in the\r\n          front row of a Classroom it reads: Martin Luther King in a\r\n          Communist Training School.\r\n                         \r\n                                   BEAUREGARD- KLAN NARRATOR (CONT\'D)\r\n                       ...and their Army of Commies started\r\n                       their Civil Rights Assault on our\r\n                       Holy White Protestant Values.\r\n                         \r\n          CLOSE - BOUREGARD - KLAN NARRATOR\r\n                         \r\n                                   BEAUREGARD- KLAN


BEAUREGARD- KLAN NARRATOR (CONT\'D)\r\n                       Do you really want your precious\r\n                       White Child going to School with\r\n                       Negroes?\r\n                         \r\n          Footage of Black and White Children playing together,\r\n          innocent.\r\n                         \r\n          Beauregard now stands by a Large Screen and points at The\r\n          Screen.\r\n                         \r\n                                   BEAUREGARD-KLAN NARRATOR (CONT\'D)\r\n                       They are Lying, Dirty


They are Lying, Dirty Monkeys...\r\n                         \r\n          FOOTAGE and STILLS of Stereotype Blacks Coons, Bucks and\r\n          shining Black Mammies. Black Soldiers in D. W. Griffith\'s\r\n          "Birth of a Nation" pushing Whites around on the Street.\r\n                         \r\n          CLOSE - BEAUREGARD\r\n                         \r\n                                   BEAUREGARD- KLAN NARRATOR (CONT\'D)\r\n                       ...Stopping at nothing to gain\r\n                       Equality with The White Man.\r\n                         \r\n


\r\n          Images and Scientific charts of Blacks compared to Apes and\r\n          Monkeys.\r\n                         \r\n          CLOSE - BEAUREGARD - KLAN NARRATOR\r\n                         \r\n                                   BEAUREGARD- KLAN NARRATOR (CONT\'D)\r\n                       ...Rapists, Murderers...Craving The\r\n                       Virgin, Pure Flesh of White Women.\r\n                       They are Super Predators...\r\n                                                                             CUT TO:\r\n


CUT TO:\r\n                         \r\n          LYNCH, The MULATTO, lusting after our LILLIAN GISH in "Birth\r\n          of a Nation." Other Lusting Images of Craving Black\r\n          Beasts!!! SEXUAL PREDATORS!!!\r\n                         \r\n                                                                             CUT TO:\r\n                         \r\n          KING KONG on Empire State Building with Fay Wray in his hand.\r\n          GUS in "Birth of a Nation" chasing a White Woman he wants to\r\n          Rape.\r\n


\r\n                                                                             CUT TO:\r\n                         \r\n          CLOSE - BEAUREGARD - KLAN NARRATOR\r\n                         \r\n          A Stereotype illustration of Jews controlling Negroes.\r\n                         \r\n                                   BEAUREGARD- KLAN NARRATOR (CONT\'D)\r\n                       ...and the Negro\'s insidious tactics\r\n                       under the tutelage of High Ranking\r\n


of High Ranking\r\n                       Blood Sucking Jews! Using an Army of\r\n                       outside...\r\n                         \r\n          Beauregard continues.\r\n                         \r\n                                                                             CUT TO:\r\n                         \r\n                                   BEAUREGARD-KLAN NARRATOR(CONT\'D)\r\n                       ...Northern Black Beast Agitators...\r\n                         \r\n          Footage of The March on


Footage of The March on Washington.\r\n                         \r\n                                                                             CUT TO:\r\n                         \r\n          CLOSE - BOUREGARD - KLAN NARRATOR.\r\n                         \r\n                                   BOUREGARD- KLAN NARRATOR (CONT\'D)\r\n                       ...determined to overthrow The God\r\n                       Commanded and Biblically inspired\r\n                       Rule of The White Race.\r\n


\r\n                                                                             CUT TO:\r\n                         \r\n          An image of an All-American White Nuclear Family.\r\n                         \r\n                                                                             CUT TO:\r\n                         \r\n          Bouregard gives his Final Words.\r\n                         \r\n                                   BOUREGARD-KLAN


BOUREGARD-KLAN NARRATOR (CONT\'D)\r\n                       It\'s an International... Jewish...\r\n                       Conspiracy.\r\n          WE HEAR and end with the Corny Stinger of Music that goes\r\n          with these Education and Propaganda Films!\r\n                         \r\n                                                                             CUT TO:\r\n                         \r\n          EXT. COLORADO SPRINGS AREA - DAY\r\n                         \r\n                         DRONE SHOT\r\n


SHOT\r\n                         \r\n          Superimposed: Early 70s\r\n                         \r\n          An amazing contrast. The beautiful landscape of Colorado\r\n          Springs, the City sits nestled within the rugged Mountain\r\n          terrain. The majestic Pikes Peak, the jagged beauty of The\r\n          Garden of the Gods, The plush Broadmoor Resort, The Will\r\n          Rodgers Shrine of The Sun.\r\n                         \r\n                         \r\n          EXT. COLORADO SPRINGS STREET - DAY\r\n                         \r\n          RON STALLWORTH, Black, 21, Handsome, Intelligent, sporting a\r\n          good


sporting a\r\n          good sized Afro, rebellious but straight laced by most 1970\'s\r\n          standards.\r\n                         \r\n          Ron stares at an Ad attached to a bulletin board.\r\n                         \r\n          CLOSE - THE AD READS:\r\n                         \r\n          JOIN THE COLORADO SPRINGS POLICE FORCE, MINORITIES ENCOURAGED\r\n          TO APPLY! Ron rips the Ad from the board.\r\n                         \r\n          EXT. COLORADO SPRINGS POLICE DEPT BUILDING. - DAY\r\n                         \r\n          INT. OFFICE OF CHIEF BRIDGES - COLORADO SPRINGS POLICE DEPT -\r\n


- COLORADO SPRINGS POLICE DEPT -\r\n                         DAY\r\n                         \r\n          A drab, white-walled office. Ron sits across the table from\r\n          The Assistant City Personnel Manager, MR. TURRENTINE, Black,\r\n          40\'s, business like but progressive and CHIEF BRIDGES, White,\r\n          smart, 50\'s, in a Police Uniform, a Man ready for change.\r\n                         \r\n                                    MR. TURRENTINE\r\n                       Why weren\'t you drafted into the\r\n                       Vietnam War?\r\n                         \r\n


\r\n                                                  RON STALLWORTH\r\n                       I went to College.\r\n                         \r\n                                   MR. TURRENTINE\r\n                       How do you feel about Vietnam?\r\n                         \r\n                                                  RON STALLWORTH\r\n                       I have mixed feelings.\r\n                                     CHIEF BRIDGES\r\n          Would you call yourself a Womanizer?\r\n


Would you call yourself a Womanizer?\r\n                                                  RON STALLWORTH\r\n                       No Sir, I would not.\r\n                         \r\n                                   MR. TURRENTINE\r\n                       Do you frequent Night Clubs?\r\n                         \r\n                                                  RON STALLWORTH\r\n                       No Sir.\r\n                         \r\n


CHIEF BRIDGES\r\n                       Do you drink?\r\n                         \r\n                                                  RON STALLWORTH\r\n                       On Special occasions, Sir.\r\n                         \r\n                                   MR. TURRENTINE\r\n                       Have you ever done any Drugs?\r\n                         \r\n                                                  RON STALLWORTH\r\n


Only those prescribed by My Doctor,\r\n                       Sir.\r\n                         \r\n          Turrentine looks at Chief Bridges.\r\n                         \r\n                                   MR. TURRENTINE\r\n                       That\'s kind of rare these days for a\r\n                       young Hip Soul Brother like you.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       I know but my Father was in The\r\n


but my Father was in The\r\n                       Military and I was raised up the\r\n                       Right way, Sir.\r\n                         \r\n                                                  CHIEF BRIDGES\r\n                       How are you with people, generally?\r\n                         \r\n                                                  RON STALLWORTH\r\n                       Sir, they treat me right, I treat\r\n                       them right, like I already said I was\r\n


raised...\r\n                         \r\n                                                  CHIEF BRIDGES\r\n                       ...Have you ever had any negative...\r\n                         \r\n          Mr. Turrentine jumps in, impatient.\r\n                         \r\n                                   MR. TURRENTINE\r\n                       ...What would you do if another Cop\r\n                       called you a Nigger?\r\n                         \r\n


RON STALLWORTH\r\n                       Would that happen...\r\n                         \r\n                                   MR. TURRENTINE\r\n                       ...Sheeeeeeettt!!!\r\n          Bridges looks at him. Turrentine waits, Ron doesn\'t know how\r\n          to respond, finally. Turrentine leans forward.\r\n                         \r\n                                   MR. TURRENTINE (CONT\'D)\r\n                       There\'s never been a Black Cop in\r\n                       this City. If we


this City. If we make you an Officer,\r\n                       you would, in effect, be the Jackie\r\n                       Robinson of the Colorado Springs\r\n                       Police force.\r\n                         \r\n          Mr. Turrentine lets this sink in.\r\n                         \r\n                                   MR. TURRENTINE (CONT\'D)\r\n                       And if you know anything about Jackie\r\n                       Robinson you know he had to take a\r\n                       lot of... guff... from his fellow\r\n


his fellow\r\n                       Teammates, from Fans, other Teams,\r\n                       and The Press.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       I know Jackie\'s Story, Sir.\r\n                         \r\n                                   MR. TURRENTINE\r\n                       Good. So, knowing that, when someone\r\n                       calls you Nigger will you be able to\r\n                       turn the other Cheek?\r\n


\r\n          Ron evaluates the hard reality of the question. Decides.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       If I need to, yes, Sir.\r\n                         \r\n                                   MR. TURRENTINE\r\n                       Son, The Mayor and I think you might\r\n                       be The Man to open things up here.\r\n                         \r\n          Ron looks at Chief Bridges.\r\n


Bridges.\r\n                         \r\n                                                  CHIEF BRIDGES\r\n                       I\'ll have your back but I can only do\r\n                       so much. The Weight of this is on\r\n                       You...and You alone.\r\n                         \r\n          Ron weighs The Journey ahead.\r\n                         \r\n                         OMITTED\r\n                         \r\n                         OMITTED\r\n


\r\n          INT. RECORDS ROOM - CSPD - DAY\r\n                         \r\n          Ron sorts a file cabinet of records as OFFICER CLAY MULANEY,\r\n          60\'s, White, sits on a stool, reading a Magazine clearly\r\n          looking at a Photo of something good.\r\n          Ron looks at the Photo of the Actress Cybill Shepherd.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       Cybill Shepherd. She was great in The\r\n                       Last Picture Show.\r\n


\r\n                                                  OFFICER MULANEY\r\n                       Never saw it but what you think?\r\n                         \r\n                                                  RON STALLWORTH\r\n                       She\'s a very good Actress.\r\n                         \r\n                                                  OFFICER MULANEY\r\n                       Y\'know you want some of that.\r\n


\r\n          Ron ignores it.\r\n                         \r\n                                   OFFICER MULANEY (CONT\'D)\r\n                       Truth be told when I see one of your\r\n                       kind with a White Woman it turns my\r\n                       Stomach.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       Yeah. Why\'s that?\r\n                         \r\n


OFFICER MULANEY\r\n                       He could only want one thing.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       What would that be?\r\n                         \r\n                                                  OFFICER MULANEY\r\n                       You like acting dumb, Y\'know.\r\n                         \r\n


RON STALLWORTH\r\n                       No, I just like my questions to be\r\n                       answered.\r\n                         \r\n          A VOICE of UNIFORMED COP WHEATON calls from the other side of\r\n          the Counter.\r\n                         \r\n                                   WHEATON (O.S.)\r\n                       Hey! Anybody in there? Looking for a\r\n                       Toad here.\r\n                         \r\n          Ron walks to the Counter to see The White and


Ron walks to the Counter to see The White and sleep-deprived\r\n          Cop impatiently leaning on his elbows.\r\n                         \r\n                                                  WHEATON (CONT\'D)\r\n                       Get me the record for this Toad named\r\n                       Tippy Birdsong.\r\n                         \r\n          Ron pulls up the File for Tippy Birdsong. The Photo shows a\r\n          Black Man in his twenties.\r\n                                                  WHEATON (CONT\'D)\r\n                       While you\'re at it, why


While you\'re at it, why don\'t you\r\n                       grab another Toad... Steven Wilson.\r\n                         \r\n          Ron pulls the File... another young Black Male, ANOTHER\r\n          SEXUAL PREDATOR!\r\n                         \r\n          INT. CSPD HALLWAY - DAY\r\n                         \r\n          Chief Bridges strides down the hall with SGT. TRAPP a soft-\r\n          spoken White Man in his 40\'s, they are discussing a File. Ron\r\n          suddenly appears walking with them.\r\n                         \r\n


RON STALLWORTH\r\n                       While I\'ve got you both here. Sirs,\r\n                       I\'d like to be an Undercover\r\n                       Detective.\r\n                         \r\n          Chief Bridges and Sgt. Trapp both stop.\r\n                         \r\n                                                  CHIEF BRIDGES\r\n                       What Narcotics?\r\n                         \r\n                                                  RON


RON STALLWORTH\r\n                       Whatever Department works, Sir.\r\n                         \r\n                                   SGT. TRAPP\r\n                       You just joined The Force, Rookie.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       I know, Sir but I think I could do\r\n                       some good there.\r\n                         \r\n                                   SGT. TRAPP\r\n


SGT. TRAPP\r\n                       Is that right?\r\n                         \r\n                                                  RON STALLWORTH\r\n                       Well, I\'m young. I think there\'s a\r\n                       niche for me. Get In where I can Fit\r\n                       In.\r\n                         \r\n                                   SGT. TRAPP\r\n                       What do you think, Chief?\r\n                         \r\n          Sgt. Trapp sees


Sgt. Trapp sees the logic, looks to Chief Bridges, who stops,\r\n          considering.\r\n                         \r\n                                                  CHIEF BRIDGES\r\n                       Think a lot of yourself, don\'t cha?\r\n                         \r\n                                                  RON STALLWORTH\r\n                       Just trying to be of help, Chief.\r\n                       Plus, I hate working in The Records\r\n                       room.\r\n          Sgt. Trapp reacts knowing Ron


Sgt. Trapp reacts knowing Ron shouldn\'t have said that about\r\n          the Records Room. CHIEF BRIDGES looks at Ron, matter of fact.\r\n                         \r\n                                                  CHIEF BRIDGES\r\n                       Well, I think Records is a good place\r\n                       for you to start, Rookie.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       Chief, want me clean shaven?\r\n                         \r\n


\r\n                                                  CHIEF BRIDGES\r\n                       Keep it. I like the look.\r\n                         \r\n          Chief Bridges walks off without another word. SGT. TRAPP\r\n          gives a knowing look to Ron, who watches them walk away.\r\n                         \r\n          INT. RECORDS ROOM - CSPD - DAY\r\n                         \r\n          Ron behind the Counter. MASTER PATROLMAN ANDY LANDERS, White,\r\n          Mid-30\'s, a regular guy but there is something dangerous\r\n          there, steps up.\r\n


\r\n                                                  LANDERS\r\n                       Need a File on a Toad.\r\n                         \r\n          Ron doesn\'t respond.\r\n                         \r\n                                                  LANDERS (CONT\'D)\r\n                       You Deaf? I said I need info on a\r\n                       Toad.\r\n                         \r\n