Upload folder using huggingface_hub

README.md

@@ -1,156 +1,75 @@
 ---
 tags:
-- generated_from_trainer
+- physics
+- cosmology
 model-index:
-- name: home/005/th5351/output
+- name: cosmosage-v3
   results: []
+license: mit
+language:
+- en
+pipeline_tag: text-generation
+base_model: meta-llama/meta-llama-3-8b
+datasets:
+- teknium/OpenHermes-2.5
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+# cosmosage
 
-[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
-<details><summary>See axolotl config</summary>
+cosmosage is a natural-language cosmology assistant that can answer questions about cosmology.
 
-axolotl version: `0.4.1`
-```yaml
-base_model: /home/005/th5351/models/cosmosage-llama3-8b-base/
-model_type: LlamaForCausalLM
-tokenizer_type: AutoTokenizer
+cosmosage-v3 is the latest iteration in the cosmosage series, trained on the LLAMA-3-8B base
+model. We started with continued pretraining on thousands of papers and textbooks. The next step 
+was fine-tuning on synthetically-generated question-answer pairs. In addition, the OpenHermes 2.5
+dataset was used to improve instruction following and general conversational capability.
 
-load_in_8bit: false
-load_in_4bit: false
-strict: false
+cosmosage-v3 is a full chat model, though it excels in Q&A mode, where the model gives a single
+answer in response to a single question.
 
-chat_template: llama3
-datasets:
-  - path: /home/005/th5351/datasets/combined_sft.jsonl
-    type: chat_template
-    chat_template: llama3
-    field_messages: conversations
-    message_field_role: from
-    message_field_content: value
-    roles:
-      system:
-        - system
-      user:
-        - human
-      assistant:
-        - gpt
-    
-
-dataset_prepared_path: /home/005/th5351/output/last_run_prepared
-val_set_size: 0.001
-eval_sample_packing: false
-output_dir: /home/005/th5351/output
-
-sequence_len: 8192
-sample_packing: true
-pad_to_sequence_len: true
-
-wandb_project:
-wandb_entity:
-wandb_watch:
-wandb_name:
-wandb_log_model:
-
-gradient_accumulation_steps: 8
-micro_batch_size: 1
-num_epochs: 2
-optimizer: paged_adamw_8bit
-lr_scheduler: cosine
-learning_rate: 5e-5
-cosine_min_lr_ratio: 0.2
-cosine_constant_lr_ratio: 0.8
-max_grad_norm: 3.0
-
-seed: 42
-
-train_on_inputs: false
-group_by_length: false
-bf16: auto
-fp16:
-tf32: false
-
-gradient_checkpointing: true
-early_stopping_patience:
-resume_from_checkpoint:
-logging_steps: 1
-xformers_attention:
-flash_attention: true
-
-warmup_steps: 100
-evals_per_epoch: 5
-eval_table_size:
-saves_per_epoch: 1
-debug:
-deepspeed: /home/005/th5351/packages/axolotl/deepspeed_configs/zero2.json
-ddp_timeout: 3600000
-weight_decay: 0.0
-fsdp:
-fsdp_config:
-
-```
-
-</details><br>
-
-# home/005/th5351/output
-
-This model was trained from scratch on the None dataset.
-It achieves the following results on the evaluation set:
-- Loss: nan
-
-## Model description
-
-More information needed
-
-## Intended uses & limitations
-
-More information needed
-
-## Training and evaluation data
-
-More information needed
-
-## Training procedure
-
-### Training hyperparameters
-
-The following hyperparameters were used during training:
-- learning_rate: 5e-05
-- train_batch_size: 1
-- eval_batch_size: 1
-- seed: 42
-- distributed_type: multi-GPU
-- num_devices: 4
-- gradient_accumulation_steps: 8
-- total_train_batch_size: 32
-- total_eval_batch_size: 4
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: cosine
-- lr_scheduler_warmup_steps: 100
-- num_epochs: 2
-
-### Training results
-
-| Training Loss | Epoch  | Step | Validation Loss |
-|:-------------:|:------:|:----:|:---------------:|
-| 1.3757        | 0.0005 | 1    | nan             |
-| 0.8083        | 0.1999 | 388  | nan             |
-| 0.8005        | 0.3998 | 776  | nan             |
-| 0.7389        | 0.5998 | 1164 | nan             |
-| 0.7269        | 0.7997 | 1552 | nan             |
-| 0.7069        | 0.9996 | 1940 | nan             |
-| 0.5786        | 1.1613 | 2328 | nan             |
-| 0.5385        | 1.3613 | 2716 | nan             |
-| 0.5381        | 1.5612 | 3104 | nan             |
-| 0.5273        | 1.7611 | 3492 | nan             |
-| 0.527         | 1.9610 | 3880 | nan             |
-
-
-### Framework versions
-
-- Transformers 4.41.1
-- Pytorch 2.3.0+cu121
-- Datasets 2.19.1
-- Tokenizers 0.19.1
+The code used to generate cosmosage is available at https://github.com/tijmen/cosmosage
+
+## Usage
+
+cosmosage-v3 uses the Llama-3 prompt template. Sampling parameters are up to you, but I like {'temperature': 0.7, 'smoothing_factor': 1, 'smoothing_curve': 1.5, 'repetition_penalty': 1.1}. 
+
+## Comparison to cosmosage_v2
+
+cosmosage-v3 was trained on the same data with nearly the same hyperparameters as cosmosage_v2.
+The primary difference is the use of the LLAMA-3-8B base model rather than the Mistral-v0.1 base
+model. 
+
+## Training details
+
+cosmosage-v3 was trained on 4xA100 (40 GB) at Gadi (NCI, Australia). A big thanks goes out to Yuan-Seng Ting for providing these resources. 
+
+## Example output
+
+**User:**
+> What is "stacking"?
+
+**Assistant:**
+> Stacking is a technique used in astronomy to combine multiple images or data sets of the same object into a single image or dataset. This can be done by aligning and co-adding the individual images or data points, which helps to improve the signal-to-noise ratio and reveal faint features that may not be visible in a single image or dataset. Stacking is often used to study the properties of galaxies, galaxy clusters, and other astronomical objects.
+
+**User:**
+> Explain the ISW effect.
+
+**Assistant:**
+> The Integrated Sachs-Wolfe (ISW) effect is a phenomenon in modern cosmology that describes the gravitational interaction between cosmic microwave background (CMB) photons and large-scale structures in the universe. It occurs when CMB photons pass through regions of space where the gravitational potential has changed due to the expansion of the universe or the presence of matter overdensities.
+
+When a photon moves into an area with a higher gravitational potential, it gains energy as it falls into the potential well. As the photon climbs out of this well on the other side, it loses energy. If the change in the gravitational potential is gradual over time, the photon's energy gain and loss will be integrated, leading to a net temperature shift in the CMB radiation. This effect can either cause a blueshift (increased energy) if the gravitational potential is decreasing, or a redshift (decreased energy) if the gravitational potential is increasing.
+
+The ISW effect is particularly important for understanding the evolution of the universe and the role of dark energy. The detection of the ISW effect has provided evidence for the existence of dark energy and has helped constrain its properties. By analyzing the CMB temperature fluctuations and cross-correlating them with tracers of large-scale structure such as galaxy surveys, researchers can study the ISW effect and learn more about the dynamics of the universe.
+
+**User:**
+> What is Digital Active Nulling?
+
+**Assistant:**
+> Digital Active Nulling (DAN) is a technique used in the readout of Transition Edge Sensor (TES) bolometers. It works by dynamically nulling the current flowing through the SQUID input coil while digitally demodulating the carrier voltage and the current signal. This method helps to minimize non-linearity effects due to the nonlinear response of the SQUID, improving the overall performance of the system.
+
+**User:**
+> Explain why the time of matter-radiation equality affects the CMB damping tail.
+
+**Assistant:**
+> The time of matter-radiation equality is when the energy densities of matter and radiation were equal. This has an impact on the Cosmic Microwave Background (CMB) damping tail because it determines the amount of photon diffusion that occurs before decoupling. A later time of matter-radiation equality results in more photon diffusion, leading to a smaller amplitude in the damping tail.
+
+[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)

config.json

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "/home/005/th5351/models/cosmosage-llama3-8b-base/",
+  "_name_or_path": "/home/005/th5351/models/cosmosage-llama3-8b/",
   "architectures": [
     "LlamaForCausalLM"
   ],

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0db3e5f497f3fa124ca527820461bf9624c4503bf87935b7a2bf36d4befab67a
+oid sha256:1d8d275ba30cd97e1bf6280c8ceacf4bf23d647760b7258d55a16642a14d9a8c
 size 16060563132