"ParserError: Error tokenizing data. C error: Expected 2 fields in line 4, saw 4"
Labels: name: bug

### System Info

Using Google Colab, Python 3.10.12. Installing latest version of langchain as of today (v0.0.219, I believe).

### Who can help?

@hwchase17

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

When running the following function to create a chain on a collection of text files , I get the following error:

```
def get_company_qa_chain(company_text_dir):


  loader = DirectoryLoader(company_text_dir, glob="*.txt")
  documents = loader.load()

  text_splitter = RecursiveCharacterTextSplitter(
      chunk_size = 1000,
      chunk_overlap  = 100,
      length_function = len,
  )

  docs = text_splitter.split_documents(documents)

  embeddings = OpenAIEmbeddings() # default model="text-embedding-ada-002"

  docsearch = Chroma.from_documents(docs, embeddings)

  model_name = "gpt-3.5-turbo"

  temperature=0

  chat = ChatOpenAI(model_name = model_name, temperature=temperature)


  template="""You are a helpful utility for answering questions. Communicate in English only."""

  system_message_prompt = SystemMessagePromptTemplate.from_template(template)

  human_template = """Use the following context to answer the question that follows. If you don't know the answer,
  say you don't know, don't try to make up an answer:

  {context}

  Question: {question}
  """

  human_message_prompt = HumanMessagePromptTemplate.from_template(human_template,
      input_variables=["context", "question"])

  chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])


  chain_type_kwargs = {"prompt": chat_prompt}
  qa = RetrievalQA.from_chain_type(llm=chat, chain_type="stuff",
                                  retriever=docsearch.as_retriever(),
                                  return_source_documents=True,
                                  chain_type_kwargs=chain_type_kwargs)

  return qa
```
```
/usr/local/lib/python3.10/dist-packages/pandas/_libs/parsers.pyx in pandas._libs.parsers.raise_parser_error()
ParserError: Error tokenizing data. C error: Expected 2 fields in line 4, saw 4
```

I have isolated the error to a particular text file, which has been produced by pymupdf. There are no clearly strange characters in the file, and everything has been cleaned using the following function before writing to text file:

```
def clean_str(text):
  text = unicodedata.normalize('NFKC', text)
  text = text.replace("{", "").replace("}", "").replace("[", "").replace("]", "")
  return text
```

### Expected behavior

I would expect the code to handle this error and skip the problematic file. I would be also totally happy with a way of cleaning the input files to ensure this doesn't happen again. It was quite a bit of work to hunt down the problem file manually.