Update README.md

README.md

@@ -1,6 +1,8 @@
 # Usage
 - Metharme format (Mistral works too but untested)
 
+---
+
 # Upscaled Tuning Experiment Write Up Thingy
 
 My cute attempt at being a siyantis :3 uwu ~
@@ -56,27 +58,38 @@ slices:
      34.5%           41.8%            23.7%
 ```
 
+---
+
 # How did the finetune go?
 
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/65f2fd1c25b848bd061b5c2e/uo_GtNKPZ_KaWJCoAcB92.png)
 
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/65f2fd1c25b848bd061b5c2e/SmloL6rPXe9jJuyQNpHKG.png)
 
+---
+
 # Weight Difference Visualization
 - Contorl A: Nemo x Rocinante
 - Control B: Small x Cydonia
+- Control C: Upscaled Nemo x Theia
 - Sample A: 39B Upscale x Tunguska 1 Epoch
 - Sample B: 39B Upscale x Tunguska 2 Epochs
 - Sample C: Tunguska 1 Epoch x Tunguska 2 Epochs
 
+
+***Note the layer sequence and other labels since it will be unreadable for the 39B**
+
+***Visualization Issue: Some layer types like `input_layernorm` look unchanged because the highest value (usually layer 0) dilluted the entire heatmap**
+
 ## Control A (Nemo 12B & Rocinante 12B, similar training)
-**Note the layer sequence and other labels since it will be unreadable for the 39B**
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/65f2fd1c25b848bd061b5c2e/EZN8Ci2_vAGmdq0WUyrpN.png)
 
 ## Control B (Small 22B & Cydonia 22B, similar training)
-**Visualization Issue: Some layer types like `input_layernorm` look unchanged because the highest value (usually layer 0) dilluted the entire heatmap**
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/65f2fd1c25b848bd061b5c2e/xdH_7fy9HuhSzaSE2-h4X.png)
 
+## Control C (Upscaled Nemo 21B & Theia 21B, similar training)
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/65f2fd1c25b848bd061b5c2e/RTvz5g8_fd5g8ZMLmawlv.png)
+
 ## Sample A: Tunguska 39B 1st Epoch vs. its 39B base
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/65f2fd1c25b848bd061b5c2e/X3-bHyQg03-QvZFvOhGp7.png)
 
@@ -91,7 +104,7 @@ WIP
 
 # Impressions
 - Using the same LR on a larger model should have been more destructive. In practice, the duplicated layers must have saved the upscale from becoming dumber than Cydonia. The larger model seems to have preserved more of its smarts thanks to the 'empty, extra' duplicate layers.
-- The upscale clearly has an effect on training. Control A & B show a smooth gradient unlike Sample A, B, C where the duplicated layers perturb the heatmap.
+- The upscale clearly has an effect on training. Control A & B show a smooth gradient unlike Control C & Sample A, B, C where the duplicated layers perturb the heatmap.
 - The duplicated layers seem to be playing catch up with the original layers. The change difference on the duplicate layers in Sample B show that they are more responsive to training than the original layers. There are no indications that the duplicate layers are 'slowing down', given that the gradient pattern remains the same in Sample A, B, and especially C (where C can be considered the rate of change).
 - In some layer types, the change difference for the first two layers are overwhelmingly huge compared to the other layers seeing as how they diluted the entire heatmap!
 - The duplicated layers on all layer types (except one) are extra sensitive. `post_attention_layernorm` interestingly had some changes in the upscale's duplicated layers, unlike Cydonia where latter layers were completely unchanged.