Repository: jerryjliu/llama_index
Issue: [Bug]: Explicitly imported SimpleDirectoryReader does not add extra_info to parsed nodes

### Bug Description

I am deploying an app based on Llamaindex inside of a docker container. I ran into a docker-related issue using the `download_loader()` to create a `SimpleDirectoryReader`:

``` python 
  # Create the GPTVectorStoreIndex
  logger.debug(f"Build index from {tempdir_path}")
  SimpleDirectoryReader = download_loader("SimpleDirectoryReader")
  loader = SimpleDirectoryReader(
      tempdir_path, recursive=True, exclude_hidden=False
  )
  documents = loader.load_data()
  logger.debug("Loaded...")
```

The dynamic installation and loading of `SimpleDirectoryReader` worked as expected for the most part in a local container that has the local app directory as a read/write volume. After a query, the source node looks like this:

```
Node: ['__abstractmethods__', '__annotations__', '__class__', '__dataclass_fields__', '__dataclass_params__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__match_args__', '__module__', '__ne__', '__new__', '__post_init__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '__weakref__', '_abc_impl', '_generate_doc_hash', 'child_node_ids', 'dataclass_json_config', 'doc_hash', 'doc_id', 'embedding', 'extra_info', 'extra_info_str', 'from_dict', 'from_json', 'get_doc_hash', 'get_doc_id', 'get_embedding', 'get_node_info', 'get_origin_type', 'get_text', 'get_type', 'get_types', 'is_doc_id_none', 'is_text_none', 'next_node_id', 'node_info', 'parent_node_id', 'prev_node_id', 'ref_doc_id', 'relationships', 'schema', 'text', 'to_dict', 'to_json']
Extra info: {'file_name': 'NVCA-2020-Stock-Purchase-Agreement-1-September-1-2020_eVQ4P4y.docx'}
Node info: {'start': 56846, 'end': 60980, '_node_type': '1'}
[NVCA-2020-Stock-Purchase-Agreement-1-September-1-2020.docx](https://github.com/jerryjliu/llama_index/files/11899087/NVCA-2020-Stock-Purchase-Agreement-1-September-1-2020.docx)
```

**In production,** however, my Docker container does not have write access to the python install directory or packages dir,  *but* it appears loading the `SimpleDirectoryReader` using `download_loader` in this way tries to modify packages on the fly (not a huge surprise, but I hadn't considered this ahead of time). I didn't want to have the container modifying my Python packages dynamically for a) security reason and b) this has a performance penalty as the containers will be loaded and stopped repeatedly and so this install process needs to re-occur repeatedly, increasing latency and bandwidth costs. 

I tried to work around this by manually installing `llama-hub` when creating the container and then loading `SimpleDirectoryLoader` explicitly like so:

```python
from llama_index.readers.file.base import SimpleDirectoryReader

# Create the GPTVectorStoreIndex
logger.debug(f"Build index from {tempdir_path}")
loader = SimpleDirectoryReader(
    tempdir_path, recursive=True, exclude_hidden=False
)
documents = loader.load_data()
logger.debug("Loaded...")
```

Now my docker issue is solved, there are no run time modifications of my python directory, **BUT** it appears the instantiated readers are no longer adding the `extra_info` metadata. Here's my logs related to the source nodes:

```
Node: ['__abstractmethods__', '__annotations__', '__class__', '__dataclass_fields__', '__dataclass_params__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__match_args__', '__module__', '__ne__', '__new__', '__post_init__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '__weakref__', '_abc_impl', '_generate_doc_hash', 'child_node_ids', 'dataclass_json_config', 'doc_hash', 'doc_id', 'embedding', 'extra_info', 'extra_info_str', 'from_dict', 'from_json', 'get_doc_hash', 'get_doc_id', 'get_embedding', 'get_node_info', 'get_origin_type', 'get_text', 'get_type', 'get_types', 'is_doc_id_none', 'is_text_none', 'next_node_id', 'node_info', 'parent_node_id', 'prev_node_id', 'ref_doc_id', 'relationships', 'schema', 'text', 'to_dict', 'to_json']
Extra info: None
Node info: {'start': 24890, 'end': 29195, '_node_type': '1'}
```

Any idea what's up here?



### Version

0.6.30

### Steps to Reproduce

Instantiate a SimpleDirectoryReader using download_loader() and load attached docx. `Extra_info` will be populated in returned queries' sources. 

Then, instantiate a SimpleDirectoryReader loaded via `from llama_index.readers.file.base import SimpleDirectoryReader`. Create an index with the same docx and run a query. No extra_info is attached to the response nodes. 

[NVCA-2020-Stock-Purchase-Agreement-1-September-1-2020.docx](https://github.com/jerryjliu/llama_index/files/11899174/NVCA-2020-Stock-Purchase-Agreement-1-September-1-2020.docx)


### Relevant Logs/Tracbacks

```shell
See above
```


Changed files:
llama_index/readers/file/ipynb_reader.py, llama_index/readers/file/markdown_reader.py, llama_index/readers/file/docs_reader.py, llama_index/readers/file/tabular_reader.py, llama_index/readers/file/video_audio_reader.py, llama_index/readers/file/slides_reader.py, llama_index/readers/file/epub_reader.py, llama_index/readers/file/mbox_reader.py, llama_index/readers/file/base.py, llama_index/readers/file/image_vision_llm_reader.py, llama_index/readers/file/image_reader.py, llama_index/readers/file/image_caption_reader.py

diff --git a/llama_index/readers/file/base.py b/llama_index/readers/file/base.py
index 8a3811c9..92faab7e 100644
--- a/llama_index/readers/file/base.py
+++ b/llama_index/readers/file/base.py
@@ -192,7 +192,7 @@ class SimpleDirectoryReader(BaseReader):
                     reader_cls = DEFAULT_FILE_READER_CLS[file_suffix]
                     self.file_extractor[file_suffix] = reader_cls()
                 reader = self.file_extractor[file_suffix]
-                docs = reader.load_data(input_file, metadata=metadata or {})
+                docs = reader.load_data(input_file, extra_info=metadata)
 
                 # iterate over docs if needed
                 if self.filename_as_id:
diff --git a/llama_index/readers/file/docs_reader.py b/llama_index/readers/file/docs_reader.py
index 40727945..05ab0246 100644
--- a/llama_index/readers/file/docs_reader.py
+++ b/llama_index/readers/file/docs_reader.py
@@ -13,7 +13,9 @@ from llama_index.schema import Document
 class PDFReader(BaseReader):
     """PDF parser."""
 
-    def load_data(self, file: Path, metadata: Optional[Dict] = None) -> List[Document]:
+    def load_data(
+        self, file: Path, extra_info: Optional[Dict] = None
+    ) -> List[Document]:
         """Parse file."""
         try:
             import pypdf
@@ -36,8 +38,8 @@ class PDFReader(BaseReader):
                 page_label = pdf.page_labels[page]
 
                 metadata = {"page_label": page_label, "file_name": file.name}
-                if metadata is not None:
-                    metadata.update(metadata)
+                if extra_info is not None:
+                    metadata.update(extra_info)
 
                 docs.append(Document(text=page_text, metadata=metadata))
             return docs
@@ -46,7 +48,9 @@ class PDFReader(BaseReader):
 class DocxReader(BaseReader):
     """Docx parser."""
 
-    def load_data(self, file: Path, metadata: Optional[Dict] = None) -> List[Document]:
+    def load_data(
+        self, file: Path, extra_info: Optional[Dict] = None
+    ) -> List[Document]:
         """Parse file."""
         try:
             import docx2txt
@@ -58,7 +62,7 @@ class DocxReader(BaseReader):
 
         text = docx2txt.process(file)
         metadata = {"file_name": file.name}
-        if metadata is not None:
-            metadata.update(metadata)
+        if extra_info is not None:
+            metadata.update(extra_info)
 
-        return [Document(text=text, metadata=metadata)]
+        return [Document(text=text, metadata=metadata or {})]
diff --git a/llama_index/readers/file/epub_reader.py b/llama_index/readers/file/epub_reader.py
index cb868fd8..b0967167 100644
--- a/llama_index/readers/file/epub_reader.py
+++ b/llama_index/readers/file/epub_reader.py
@@ -13,7 +13,9 @@ from llama_index.schema import Document
 class EpubReader(BaseReader):
     """Epub Parser."""
 
-    def load_data(self, file: Path, metadata: Optional[Dict] = None) -> List[Document]:
+    def load_data(
+        self, file: Path, extra_info: Optional[Dict] = None
+    ) -> List[Document]:
         """Parse file."""
         try:
             import ebooklib
@@ -38,4 +40,4 @@ class EpubReader(BaseReader):
                 )
 
         text = "\\n".join(text_list)
-        return [Document(text=text, metadata=metadata)]
+        return [Document(text=text, metadata=extra_info or {})]
diff --git a/llama_index/readers/file/image_caption_reader.py b/llama_index/readers/file/image_caption_reader.py
index ecce2f02..17092088 100644
--- a/llama_index/readers/file/image_caption_reader.py
+++ b/llama_index/readers/file/image_caption_reader.py
@@ -54,7 +54,9 @@ class ImageCaptionReader(BaseReader):
         self._keep_image = keep_image
         self._prompt = prompt
 
-    def load_data(self, file: Path, metadata: Optional[Dict] = None) -> List[Document]:
+    def load_data(
+        self, file: Path, extra_info: Optional[Dict] = None
+    ) -> List[Document]:
         """Parse file."""
         from PIL import Image
 
@@ -89,6 +91,6 @@ class ImageCaptionReader(BaseReader):
             ImageDocument(
                 text=text_str,
                 image=image_str,
-                metadata=metadata,
+                metadata=extra_info or {},
             )
         ]
diff --git a/llama_index/readers/file/image_reader.py b/llama_index/readers/file/image_reader.py
index 4fa0771f..3ef02b33 100644
--- a/llama_index/readers/file/image_reader.py
+++ b/llama_index/readers/file/image_reader.py
@@ -51,7 +51,9 @@ class ImageReader(BaseReader):
         self._keep_image = keep_image
         self._parse_text = parse_text
 
-    def load_data(self, file: Path, metadata: Optional[Dict] = None) -> List[Document]:
+    def load_data(
+        self, file: Path, extra_info: Optional[Dict] = None
+    ) -> List[Document]:
         """Parse file."""
         from PIL import Image
 
@@ -107,4 +109,6 @@ class ImageReader(BaseReader):
             # remove first task start token
             text_str = re.sub(r"<.*?>", "", sequence, count=1).strip()
 
-        return [ImageDocument(text=text_str, image=image_str, metadata=metadata)]
+        return [
+            ImageDocument(text=text_str, image=image_str, metadata=extra_info or {})
+        ]
diff --git a/llama_index/readers/file/image_vision_llm_reader.py b/llama_index/readers/file/image_vision_llm_reader.py
index 0a521623..1cbe3957 100644
--- a/llama_index/readers/file/image_vision_llm_reader.py
+++ b/llama_index/readers/file/image_vision_llm_reader.py
@@ -50,7 +50,9 @@ class ImageVisionLLMReader(BaseReader):
         self._keep_image = keep_image
         self._prompt = prompt
 
-    def load_data(self, file: Path, metadata: Optional[Dict] = None) -> List[Document]:
+    def load_data(
+        self, file: Path, extra_info: Optional[Dict] = None
+    ) -> List[Document]:
         """Parse file."""
         from PIL import Image
 
@@ -85,6 +87,6 @@ class ImageVisionLLMReader(BaseReader):
             ImageDocument(
                 text=text_str,
                 image=image_str,
-                metadata=metadata,
+                metadata=extra_info or {},
             )
         ]
diff --git a/llama_index/readers/file/ipynb_reader.py b/llama_index/readers/file/ipynb_reader.py
index 40302b6f..0e1b145b 100644
--- a/llama_index/readers/file/ipynb_reader.py
+++ b/llama_index/readers/file/ipynb_reader.py
@@ -18,7 +18,9 @@ class IPYNBReader(BaseReader):
         self._parser_config = parser_config
         self._concatenate = concatenate
 
-    def load_data(self, file: Path, metadata: Optional[Dict] = None) -> List[Document]:
+    def load_data(
+        self, file: Path, extra_info: Optional[Dict] = None
+    ) -> List[Document]:
         """Parse file."""
 
         if file.name.endswith(".ipynb"):
@@ -33,7 +35,7 @@ class IPYNBReader(BaseReader):
         splits.pop(0)
 
         if self._concatenate:
-            docs = [Document(text="\\n\\n".join(splits), metadata=metadata)]
+            docs = [Document(text="\\n\\n".join(splits), metadata=extra_info or {})]
         else:
-            docs = [Document(text=s, metadata=metadata) for s in splits]
+            docs = [Document(text=s, metadata=extra_info or {}) for s in splits]
         return docs
diff --git a/llama_index/readers/file/markdown_reader.py b/llama_index/readers/file/markdown_reader.py
index 5c600222..5f9402dd 100644
--- a/llama_index/readers/file/markdown_reader.py
+++ b/llama_index/readers/file/markdown_reader.py
@@ -99,16 +99,18 @@ class MarkdownReader(BaseReader):
         markdown_tups = self.markdown_to_tups(content)
         return markdown_tups
 
-    def load_data(self, file: Path, metadata: Optional[Dict] = None) -> List[Document]:
+    def load_data(
+        self, file: Path, extra_info: Optional[Dict] = None
+    ) -> List[Document]:
         """Parse file into string."""
         tups = self.parse_tups(file)
         results = []
         # TODO: don't include headers right now
         for header, value in tups:
             if header is None:
-                results.append(Document(text=value, metadata=metadata or {}))
+                results.append(Document(text=value, metadata=extra_info or {}))
             else:
                 results.append(
-                    Document(text=f"\\n\\n{header}\\n{value}", metadata=metadata or {})
+                    Document(text=f"\\n\\n{header}\\n{value}", metadata=extra_info or {})
                 )
         return results
diff --git a/llama_index/readers/file/mbox_reader.py b/llama_index/readers/file/mbox_reader.py
index a2639517..2219b509 100644
--- a/llama_index/readers/file/mbox_reader.py
+++ b/llama_index/readers/file/mbox_reader.py
@@ -50,7 +50,9 @@ class MboxReader(BaseReader):
         self.max_count = max_count
         self.message_format = message_format
 
-    def load_data(self, file: Path, metadata: Optional[Dict] = None) -> List[Document]:
+    def load_data(
+        self, file: Path, extra_info: Optional[Dict] = None
+    ) -> List[Document]:
         """Parse file into string."""
         # Import required libraries
         import mailbox
@@ -102,4 +104,4 @@ class MboxReader(BaseReader):
             if self.max_count > 0 and i >= self.max_count:
                 break
 
-        return [Document(text=result, metadata=metadata) for result in results]
+        return [Document(text=result, metadata=extra_info or {}) for result in results]
diff --git a/llama_index/readers/file/slides_reader.py b/llama_index/readers/file/slides_reader.py
index 01b88dbd..ca9d12c3 100644
--- a/llama_index/readers/file/slides_reader.py
+++ b/llama_index/readers/file/slides_reader.py
@@ -86,7 +86,7 @@ class PptxReader(BaseReader):
     def load_data(
         self,
         file: Path,
-        metadata: Optional[Dict] = None,
+        extra_info: Optional[Dict] = None,
     ) -> List[Document]:
         """Parse file."""
         from pptx import Presentation
@@ -110,4 +110,4 @@ class PptxReader(BaseReader):
                 if hasattr(shape, "text"):
                     result += f"{shape.text}\\n"
 
-        return [Document(text=result, metadata=metadata)]
+        return [Document(text=result, metadata=extra_info or {})]
diff --git a/llama_index/readers/file/tabular_reader.py b/llama_index/readers/file/tabular_reader.py
index bc7035a0..a9bf4484 100644
--- a/llama_index/readers/file/tabular_reader.py
+++ b/llama_index/readers/file/tabular_reader.py
@@ -27,7 +27,9 @@ class CSVReader(BaseReader):
         super().__init__(*args, **kwargs)
         self._concat_rows = concat_rows
 
-    def load_data(self, file: Path, metadata: Optional[Dict] = None) -> List[Document]:
+    def load_data(
+        self, file: Path, extra_info: Optional[Dict] = None
+    ) -> List[Document]:
         """Parse file.
 
         Returns:
@@ -44,9 +46,9 @@ class CSVReader(BaseReader):
             for row in csv_reader:
                 text_list.append(", ".join(row))
         if self._concat_rows:
-            return [Document(text="\\n".join(text_list), metadata=metadata)]
+            return [Document(text="\\n".join(text_list), metadata=extra_info)]
         else:
-            return [Document(text=text, metadata=metadata) for text in text_list]
+            return [Document(text=text, metadata=extra_info) for text in text_list]
 
 
 class PandasCSVReader(BaseReader):
@@ -91,7 +93,9 @@ class PandasCSVReader(BaseReader):
         self._row_joiner = row_joiner
         self._pandas_config = pandas_config
 
-    def load_data(self, file: Path, metadata: Optional[Dict] = None) -> List[Document]:
+    def load_data(
+        self, file: Path, extra_info: Optional[Dict] = None
+    ) -> List[Document]:
         """Parse file."""
         df = pd.read_csv(file, **self._pandas_config)
 
@@ -101,7 +105,11 @@ class PandasCSVReader(BaseReader):
 
         if self._concat_rows:
             return [
-                Document(text=(self._row_joiner).join(text_list), metadata=metadata)
+                Document(
+                    text=(self._row_joiner).join(text_list), metadata=extra_info or {}
+                )
             ]
         else:
-            return [Document(text=text, metadata=metadata) for text in text_list]
+            return [
+                Document(text=text, metadata=extra_info or {}) for text in text_list
+            ]
diff --git a/llama_index/readers/file/video_audio_reader.py b/llama_index/readers/file/video_audio_reader.py
index 459c1224..b783992d 100644
--- a/llama_index/readers/file/video_audio_reader.py
+++ b/llama_index/readers/file/video_audio_reader.py
@@ -35,7 +35,9 @@ class VideoAudioReader(BaseReader):
 
         self.parser_config = {"model": model}
 
-    def load_data(self, file: Path, metadata: Optional[Dict] = None) -> List[Document]:
+    def load_data(
+        self, file: Path, extra_info: Optional[Dict] = None
+    ) -> List[Document]:
         """Parse file."""
         import whisper
 
@@ -59,4 +61,4 @@ class VideoAudioReader(BaseReader):
 
         transcript = result["text"]
 
-        return [Document(text=transcript, metadata=metadata)]
+        return [Document(text=transcript, metadata=extra_info or {})]