Pytorch No Output from Fine Tuning

Anudit: <p>Hello Everyone,</p>
<p>I am trying to fine tune llama3 8B with custom data set in JSON format. Upon executing the command, I see that the logs file is empty, and there is nothing generated in the output directories and there is no error as well.</p>
<p>PFB my config that I am using and the command being used to run. The JSON format being followed is alpaca</p>
<p>Can you please help in letting me know where I am going wrong:</p>
<p>tune run lora_finetune_single_device --config torchtune_llama3.yaml</p>
<blockquote>
<h1>Config for single device LoRA finetuning in lora_finetune_single_device.py</h1>
<h1>using a Llama3 8B model</h1>
<h1></h1>
<h1>This config assumes that you’ve run the following command before launching</h1>
<h1>this run:</h1>
<h1>tune download meta-llama/Meta-Llama-3-8B --output-dir /tmp/Meta-Llama-3-8B --hf-token &lt;HF_TOKEN&gt;</h1>
<h1></h1>
<h1>To launch on a single device, run the following command from root:</h1>
<h1>tune run lora_finetune_single_device --config llama3/8B_lora_single_device</h1>
<h1></h1>
<h1>You can add specific overrides through the command line. For example</h1>
<h1>to override the checkpointer directory while launching training</h1>
<h1>you can run:</h1>
<h1>tune run lora_finetune_single_device --config llama3/8B_lora_single_device checkpointer.checkpoint_dir=&lt;YOUR_CHECKPOINT_DIR&gt;</h1>
<h1></h1>
<h1>This config works only for training on single device.</h1>
<h1>Model Arguments</h1>
<p>model:<br>
<em>component</em>: torchtune.models.llama3.lora_llama3_8b<br>
lora_attn_modules: [‘q_proj’, ‘v_proj’]<br>
apply_lora_to_mlp: False<br>
apply_lora_to_output: False<br>
lora_rank: 8<br>
lora_alpha: 16</p>
<h1>Tokenizer</h1>
<p>tokenizer:<br>
<em>component</em>: torchtune.models.llama3.llama3_tokenizer<br>
path: C:/GIT_External/Meta-Llama-3-8B/8B/tokenizer.model</p>
<p>checkpointer:<br>
<em>component</em>: torchtune.utils.FullModelMetaCheckpointer<br>
checkpoint_dir: C:/GIT_External/Meta-Llama-3-8B/8B/<br>
checkpoint_files: [<br>
consolidated.00.pth<br>
]<br>
recipe_checkpoint: null<br>
output_dir: C:/GIT_External/Meta-Llama-3-8B/8B/<br>
model_type: LLAMA3<br>
resume_from_checkpoint: False</p>
<h1>Dataset and Sampler</h1>
<p>dataset:<br>
<em>component</em>: torchtune.datasets.alpaca_dataset<br>
train_on_input: True<br>
format: json<br>
data_path: C:/Products/GenAI/Training/testtrain.json<br>
seed: null<br>
shuffle: True<br>
batch_size: 2</p>
<h1>Optimizer and Scheduler</h1>
<p>optimizer:<br>
<em>component</em>: torch.optim.AdamW<br>
weight_decay: 0.01<br>
lr: 3e-4<br>
lr_scheduler:<br>
<em>component</em>: torchtune.modules.get_cosine_schedule_with_warmup<br>
num_warmup_steps: 100</p>
<p>loss:<br>
<em>component</em>: torch.nn.CrossEntropyLoss</p>
<h1>Training</h1>
<p>epochs: 1<br>
max_steps_per_epoch: null<br>
gradient_accumulation_steps: 64<br>
compile: False</p>
<h1>Logging</h1>
<p>output_dir: C:/Products/GenAI/Training/Logs/<br>
metric_logger:<br>
<em>component</em>: torchtune.utils.metric_logging.DiskLogger<br>
log_dir: ${output_dir}<br>
log_every_n_steps: 1</p>
<h1>Environment</h1>
<p>device: cpu<br>
dtype: fp32<br>
enable_activation_checkpointing: True</p>
<h1>Profiler (disabled)</h1>
<p>profiler:<br>
<em>component</em>: torchtune.utils.profiler<br>
enabled: False</p>
</blockquote>

Anudit: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Can you please check and highlight what could be causing this issue?</p>