## Other Setups
SPARK pipelines are modular and we allow adjustment of many parameters and configurations.
Please find the detailed parameter explanation and the output directory hierarchies in this file.


## 1.1 SPRAK Idea Generation
  The general view of how the input and output folders are placed in default settings.

  ```bash
  output_B/
    ├──IDEA_ARCHIVE_DIR/
          ├── ideas_ProcessUID_IdeasEachCycle_timestamp1_temp.json #ideas generated and surpassed duplicate checks  in the first run are stored in this file 
          ├── ideas_ProcessUID_IdeasEachCycle_timestamp1_final.json #ideas generated and surpassed duplicate checks  in the first run are stored in this file (with structured json format)
          ├── ideas_ProcessUID_IdeasEachCycle_timestamp2_temp.json #deas generated and surpassed duplicate checks  in the second runs are stored in this file 
          └── ...

                
    ├──IDEA_DATABASE/
          ├── duplicate_idea_database_ProcessUID_timestamp.json # all ideas and their similarity scores to other ideas in the database are stored. This keeps updating as the runs go on
          └── idea_database_ProcessUID_timestamp.json # the database where all qualified ideas are stored. This keeps updating as the runs go on

    ├─IDEA_TEMP_DIR/ #input files
          ├── idea_database_ProcessUID_timestamp_REFINE.json
          └── ...


    └──LOGS/
          ├── log_ProcessUID_timestamp.csv #information of each cycle, including idea quality, number of cell types, cycle number etc.
          ├── metadata_ProcessUID_timestamp.json # metadata of the running process: LLM used, UID, predefined idea quality etc.
          ├── plot_ProcessUID_timestamp.png #The plots for counts (number of ideas added, duplicated, rejected and accumulated), similarity scores in different runs, and token usages.
          └── ProcessUID_timestamp # one line containing the index of current idea quality, current cycle, and the UID separated by comma. NOTE! this file is read by the idea generation pipeline to resume the process if set for "CHECKPOINT" in the config file. Please see detailed explanation for "CHECKPOINT" in the following section.  

  ```

  Here we illustrate with the idea generation use case 1, the idea generation scripts for other use cases work the same way in principle.

  **Configure your idea generation:** configure the model, output path, number of cell types etc.

  - **General parameter configuration**
      ```bash
        config_IG_B.json
      ```
      - `LOGS_DIR: [str]`  the path where the logs are stored
      - `IDEA_TEMP_DIR: [str]` the path to the temporary files for idea generation
      - `IDEA_ARCHIVE_DIR: [str]` the location where the intermediate and final json formatted ideas for each cycle of generation are stored
      - `AGENT_VERBOSITY: [boolean]` switch on/off of verbosity for agent activities
      - `TASK_VERBOSITY: [boolean]` switch on/off of verbosity for task descriptions
      - `CREW_VERBOSITY: [boolean]` switch on/off of verbosity for crewai activities
      - `TASK_ATTEMPT_N: [int]`: How many attempts are allowed for the given task (idea generation)
      - `CANCER_TYPE: [str]` The types of cancer for investigation
      - `NUMBER_IDEAS: [int]` How many ideas are generated for each cycle
      - `NUMBER_CYCLES: [int]` How many cycles the task should be run
      - `NUMBER_CELL_TYPES: [str]` Numbers of different cell types 
      - `IDEA_DATABASE_CHUNK_SIZE: [int]` how many ideas are read from the idea database at a time
      - `SIMILARITY_REJECTION_THRESHOLD: [int]` the similarity score (1-10) for rejecting an idea
      - `LLM_IGA: [str]`: the LLM model used for idea generation (If you are using a public model, please refer to the name, including the prefix in huggingface)
      - `LLM_IGA_CREATIVITY: [float]` temperature argument for openai models (0-1). A higher value (closer to 1) produces more diverse and creative ideas, possibly more unusual or novel.
      - `LLM_IRA: [str]` the LLM model used for idea refinement (If you are using a public model, please refer to the name, including the prefix in huggingface)
      - `LLM_DDA: [str]` the LLM model used for duplicate detection (If you are using a public model, please refer to the name, including the prefix in huggingface)
      - `LLM_IRA_DDA_CREATIVITY: [float/int]`: similar to `LLM_IGA_CREATIVITY`. By default, it is set to **99** which is a special convention for o3-mini that do not have a native temperature parameter.
      - `LLM_REASONING_EFFORT: [str]` reasoning effort for idea generation, refinement and duplicate detection 
      - `IDEA_QUALITY_LIST: [list]` a list of idea quality types. The idea generation will cycle the number of idea quality types and generate `NUMBER_CYCLES` for each quality type, with `NUMBER_IDEAS` in each cycle.
      - `CHECKPOINT: [str]` empty string by default. Enter the `**ProcessUID**` for connecting to the latest checkpoint. This checkpoint reads the file stored in  `/LOGS/**ProcessUID**` to resume the process from the last checkpoint.

      
  - **Agent description**
    ```bash
        ./config/agent.yaml
      ```
      - `idea_generator_agent` , `idea_reviewer_agent`, `idea_refiner_agent`, `duplicate_detector_agent`, `idea_refiner_agent`: The agent titles which are referred during agent-calling in `main_IG_B_UC1.py`. 
      - `role`, `backstory`: description of the agents' role and the backstory

  - **Task description**
    ```bash
        ./config/tasks.yaml
      ```
      - `description` `expected_output`: the description prompt which is utilized by the agent for idea generation, yielding the output in the `expected_output` format. 

  - **Task initiation**
      ```bash
        agent_task_init.yaml
      ```
      - `initialize_LLM_IG` is used for locally-deployed LLMs. `base_url` should be provided in the format `http://IP.Address:port/v1` (replace `IP.Address` and `port` with the IP address and port number for the deployed LLM).
      - `initialize_LLM` is used for openai models with API keys.

  **NOTE** provider should remain **"openai"** no matter which model is used.

## 1.2 SPRAK Idea Refinement

  **Configure your idea refinement:** configure the model, output path, number of cell types etc.

  **General parameter configuration**
  ```bash
    config_IREF.json
  ```
  (Some similar parameters are omitted for brevity. Please find them in previous sections)
  - `DATA_TYPE [str]`: it can be either `mplex` for multiplex anlysis, or `he` for H&E setup.

## 1.3 SPRAK Idea Coding

  The general view of how the input and output folders are placed in default settings.
  ```bash
  coding_inout_output/
    ├── CODE_OUTPUT/
          ├──idea_database_ProcessUID_timestamp_REFINE/
                ├── CODE_1.V0f.json #code generation failed for the first attempt
                ├── CODE_1.V1s.json #code generation succeeded for the second attempt
                ├── CODE_2.V0s.json #code generation succeeded for the first attempt
                └── ...
          ├──idea_database_small_selection_of_ideas/
                ├── CODE_1.py #ready-to-run python code snippets
                ├── CODE_2.py
                └── ...

                
    ├──CODE_WORKING_DIR/
          └──idea_database_ProcessUID_timestamp_REFINE/ working directory for the code generation pipeline

    ├─IDEA_DATABASE_CODING_INPUT/ #input files
          ├── idea_database_ProcessUID_timestamp_REFINE.json
          └── ...


    └──Small_Test_Timelog/
          ├── CODE_1.V0s.txt #Analysis finished in 0.35 sec
          ├── CODE_2.V0s.txt 
          ├── CODE_3.V0f.txt #Error: runtime threshold for code3. Implement faster computation.
          └── ...

  ```
  **Configure your idea refinement:** configure the model, output path, number of cell types etc.


  ```bash
    config_CODING.json
  ```
  - `CONDA_ENV: [str]` the name of the environment for testing the generated python code on single cell evaluation. It is by default `spark_feature_eval` (Please see [Installation Guide](./Installation/INSTALLATION.md))
 - `CONDA_SOURCE_CALL: [str]`: the command line for calling conda e.g. `source ~/miniconda3/etc/profile.d/conda.sh`
 - `CELL_PIPELINE_SCRIPT_PATH: [str]` the path to the single cell evaluation. **NOTE** We use the single cell evaluation for testing the python codes with the single cell results. Only the codes that are successfully executed and finished are qualitfied as validated code.