Repository: expectedparrot/edsl

Issue:
Modify `ScenarioList.from_directory()` to prompt user to provide a key
We have an EDSL object called "ScenarioList" that is a list of scenarios (`Scenario`). This can be send to a language model. Some models support taking files as `FileStore`, in which case our syntax is `Scenario({'content': <filestore object>})`
The ScenarioList has a classmethod to create a ScenarioList by reading the contents of a directory. However, it is returning the list of FilesStores rather than the nested structure described above. 

We want to modify it so that the user specifies a key & a directory and then even Scenario entry has that key and the unique FileStore object as the value.
The user could then use it like so: 
```
from edsl import ScenarioList

s = ScenarioList.from_directory("images_folder", key_name = "content")

q = QuestionFreeText(
question_name = "test",
question_text = "Please describe this image/text: {{ scenario.content }}"
)

r = q.by(s).run()
```
# TODO
1. Modify ScenarioList to have an optional key_name parameter 
2. If blank, use "content" 
3. Once change is made, check both unit tests & doctests.

Discussion:

Solution patch:
diff --git a/docs/filestore.rst b/docs/filestore.rst
index 3ba466821..4170fc30e 100644
--- a/docs/filestore.rst
+++ b/docs/filestore.rst
@@ -105,6 +105,7 @@ This is equivalent:
 Once retrieved, a file can be converted into scenarios.
 To construct a single scenario from a file, use the `Scenario` constructor and pass the file as a value for a specified key (see image file example below).
 To construct a list of scenarios from a file, call the `from_csv` or `from_pdf` method of the `ScenarioList` constructor and pass the file as an argument (see CSV and PDF examples below).
+To construct a list of scenarios from multiple files in a directory, you can use the `ScenarioList.from_directory()` method, which will wrap each file in a Scenario with a specified key (default is "content").
 
 We can also create agents by calling the `from_csv()` method on an `AgentList` object.
 
diff --git a/docs/scenarios.rst b/docs/scenarios.rst
index 7aeac4d09..144a7257b 100644
--- a/docs/scenarios.rst
+++ b/docs/scenarios.rst
@@ -128,6 +128,8 @@ Special methods are available for creating a `Scenario` or `ScenarioList` from v
 
 * The constructor method `from_pdf()` can be used to create a single scenario for a PDF or a scenario list where each page of a PDF is stored as an individual scenario.
 
+* The constructor method `from_directory()` can be used to create a scenario list from all files in a directory, where each file is wrapped in a Scenario object with a specified key (default is "content").
+
 * The constructor methods `from_list()`, `from_csv`, `from_nested_dict()` and `from_wikipedia_table()` will create a scenario list from a list, CSV, nested dictionary or Wikipedia table.
 
 For example, the following code will create the same scenario list as above:
@@ -137,6 +139,28 @@ For example, the following code will create the same scenario list as above:
   from edsl import ScenarioList
 
   scenariolist = ScenarioList.from_list("activity", ["running", "reading"])
+  
+Example of creating a scenario list from files in a directory:
+
+.. code-block:: python
+
+  from edsl import ScenarioList, QuestionFreeText
+  
+  # Create a ScenarioList from all image files in a directory
+  # Each file will be wrapped in a Scenario with key "content"
+  scenarios = ScenarioList.from_directory("images_folder/*.png")
+  
+  # Or specify a custom key name
+  scenarios = ScenarioList.from_directory("images_folder", key_name="image")
+  
+  # Create a question that uses the scenario key
+  q = QuestionFreeText(
+      question_name="image_description",
+      question_text="Please describe this image: {{ scenario.image }}"
+  )
+  
+  # Run the question with the scenarios
+  results = q.by(scenarios).run()
 
 
 Examples for each of these methods is provided below, and in `this notebook <https://www.expectedparrot.com/content/44de0963-31b9-4944-a9bf-508c7a07d757>`_.
diff --git a/edsl/scenarios/scenario_list.py b/edsl/scenarios/scenario_list.py
index 22bdfe0f3..48439e95f 100644
--- a/edsl/scenarios/scenario_list.py
+++ b/edsl/scenarios/scenario_list.py
@@ -895,11 +895,13 @@ def from_directory(
         cls,
         path: Optional[str] = None,
         recursive: bool = False,
+        key_name: str = "content",
     ) -> "ScenarioList":
-        """Create a ScenarioList of FileStore objects from files in a directory.
+        """Create a ScenarioList of Scenario objects from files in a directory.
         
-        This method scans a directory and creates a FileStore object for each file found,
-        optionally filtering files based on a wildcard pattern. If no path is provided,
+        This method scans a directory and creates a Scenario object for each file found,
+        where each Scenario contains a FileStore object under the specified key.
+        Optionally filters files based on a wildcard pattern. If no path is provided,
         the current working directory is used.
         
         Args:
@@ -910,25 +912,27 @@ def from_directory(
                  - "/path/to/directory/*.py" - scans only Python files in the directory
                  - "*.txt" - scans only text files in the current working directory
             recursive: Whether to scan subdirectories recursively. Defaults to False.
+            key_name: The key to use for the FileStore object in each Scenario. Defaults to "content".
             
         Returns:
-            A ScenarioList containing FileStore objects for all matching files.
+            A ScenarioList containing Scenario objects for all matching files, where each Scenario
+            has a FileStore object under the specified key.
             
         Raises:
             FileNotFoundError: If the specified directory does not exist.
             
         Examples:
-            # Get all files in the current directory
+            # Get all files in the current directory with default key "content"
             sl = ScenarioList.from_directory()
             
-            # Get all Python files in a specific directory
-            sl = ScenarioList.from_directory('*.py')
+            # Get all Python files in a specific directory with custom key "python_file"
+            sl = ScenarioList.from_directory('*.py', key_name="python_file")
             
             # Get all image files in the current directory
-            sl = ScenarioList.from_directory('*.png')
+            sl = ScenarioList.from_directory('*.png', key_name="image")
             
             # Get all files recursively including subdirectories
-            sl = ScenarioList.from_directory(recursive=True)
+            sl = ScenarioList.from_directory(recursive=True, key_name="document")
         """
         # Handle default case - use current directory
         if path is None:
@@ -1001,7 +1005,10 @@ def from_directory(
             example_suffix=example_suffix
         )
         
-        return cls(file_stores)
+        # Convert FileStore objects to Scenario objects with the specified key
+        scenarios = [Scenario({key_name: file_store}) for file_store in file_stores]
+        
+        return cls(scenarios)
                 
     @classmethod
     def from_list(


Test patch:
diff --git a/tests/scenarios/test_ScenarioListFromDirectory.py b/tests/scenarios/test_ScenarioListFromDirectory.py
index fc1a983fa..a3b384075 100644
--- a/tests/scenarios/test_ScenarioListFromDirectory.py
+++ b/tests/scenarios/test_ScenarioListFromDirectory.py
@@ -4,7 +4,7 @@
 from pathlib import Path
 import shutil
 
-from edsl.scenarios import ScenarioList, FileStore
+from edsl.scenarios import ScenarioList, FileStore, Scenario
 
 
 class TestScenarioListFromDirectory:
@@ -52,11 +52,12 @@ def test_from_directory_no_args(self, temp_directory, monkeypatch):
             
             # Should find all 6 files in the root (not recursive)
             assert len(sl) == 6
-            # All items should be FileStore instances
-            assert all(isinstance(item, FileStore) for item in sl)
+            # All items should be Scenario instances with FileStore under the "content" key
+            assert all(isinstance(item, Scenario) for item in sl)
+            assert all(isinstance(item["content"], FileStore) for item in sl)
             
             # Check if file paths are correct
-            file_paths = [os.path.basename(item["path"]) for item in sl]
+            file_paths = [os.path.basename(item["content"]["path"]) for item in sl]
             for i in range(3):
                 assert f"file{i}.txt" in file_paths
             for i in range(2):
@@ -75,8 +76,9 @@ def test_from_directory_with_path(self, temp_directory):
         
         # Should find all 6 files in the root (not recursive)
         assert len(sl) == 6
-        # All items should be FileStore instances
-        assert all(isinstance(item, FileStore) for item in sl)
+        # All items should be Scenario instances with FileStore under the "content" key
+        assert all(isinstance(item, Scenario) for item in sl)
+        assert all(isinstance(item["content"], FileStore) for item in sl)
 
     def test_from_directory_with_wildcard(self, temp_directory):
         """Test from_directory with wildcard pattern."""
@@ -85,11 +87,12 @@ def test_from_directory_with_wildcard(self, temp_directory):
         
         # Should find 2 Python files in the root
         assert len(sl) == 2
-        # All items should be FileStore instances
-        assert all(isinstance(item, FileStore) for item in sl)
+        # All items should be Scenario instances with FileStore under the "content" key
+        assert all(isinstance(item, Scenario) for item in sl)
+        assert all(isinstance(item["content"], FileStore) for item in sl)
         
         # All files should be Python files
-        suffixes = [Path(item["path"]).suffix for item in sl]
+        suffixes = [Path(item["content"]["path"]).suffix for item in sl]
         assert all(suffix == ".py" for suffix in suffixes)
 
     def test_from_directory_with_just_wildcard(self, temp_directory, monkeypatch):
@@ -105,7 +108,7 @@ def test_from_directory_with_just_wildcard(self, temp_directory, monkeypatch):
             # Should find 3 text files in the root
             assert len(sl) == 3
             # All files should be text files
-            suffixes = [Path(item["path"]).suffix for item in sl]
+            suffixes = [Path(item["content"]["path"]).suffix for item in sl]
             assert all(suffix == ".txt" for suffix in suffixes)
         finally:
             os.chdir(original_dir)
@@ -119,7 +122,7 @@ def test_from_directory_recursive(self, temp_directory):
         assert len(sl) == 8
         
         # Check if subdirectory files are included
-        file_paths = [item["path"] for item in sl]
+        file_paths = [item["content"]["path"] for item in sl]
         assert os.path.join(temp_directory, "subdir", "subfile.txt") in file_paths
         assert os.path.join(temp_directory, "subdir", "subfile.py") in file_paths
 
@@ -132,7 +135,7 @@ def test_from_directory_recursive_with_wildcard(self, temp_directory):
         assert len(sl) == 3
         
         # Check if subdirectory Python file is included
-        file_names = [os.path.basename(item["path"]) for item in sl]
+        file_names = [os.path.basename(item["content"]["path"]) for item in sl]
         assert "subfile.py" in file_names
 
     def test_from_directory_empty(self):
@@ -146,6 +149,20 @@ def test_from_directory_nonexistent(self):
         with pytest.raises(FileNotFoundError):
             ScenarioList.from_directory("/path/that/does/not/exist")
 
+    def test_from_directory_custom_key_name(self, temp_directory):
+        """Test from_directory with a custom key_name parameter."""
+        sl = ScenarioList.from_directory(temp_directory, key_name="file")
+        
+        # Should find all 6 files in the root (not recursive)
+        assert len(sl) == 6
+        # All items should be Scenario objects with FileStore under the "file" key
+        assert all(isinstance(item, Scenario) for item in sl)
+        assert all(isinstance(item["file"], FileStore) for item in sl)
+        
+        # Check if file paths are accessible through the custom key
+        file_paths = [os.path.basename(item["file"]["path"]) for item in sl]
+        assert len(file_paths) == 6
+
 
 if __name__ == "__main__":
     pytest.main()
\ No newline at end of file