ehartford commited on
Commit
9d3a0ae
1 Parent(s): 0616f84

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +13 -2
README.md CHANGED
@@ -14,6 +14,8 @@ license: apache-2.0
14
 
15
  Dolphin 2.6 Mistral 7b - DPO Laser 🐬
16
 
 
 
17
  Discord https://discord.gg/SmbBewAM
18
 
19
 
@@ -25,8 +27,17 @@ This model is based on Mistral-7b
25
 
26
  The base model has 16k context
27
 
28
- This is a special release of Dolphin-DPO.
29
- We have further carried out a noise reduction technique based on SVD decomposition, following the [paper](https://arxiv.org/pdf/2312.13558.pdf) produced @ [Microsoft Research](https://huggingface.co/microsoft/).
 
 
 
 
 
 
 
 
 
30
 
31
  We have adapted this paper on our own version of LASER, using Random Matrix Theory (Marchenko-Pastur theorem) to calculate optimal ranks instead of brute-force search.
32
 
 
14
 
15
  Dolphin 2.6 Mistral 7b - DPO Laser 🐬
16
 
17
+ By @ehartford and @fernandofernandes
18
+
19
  Discord https://discord.gg/SmbBewAM
20
 
21
 
 
27
 
28
  The base model has 16k context
29
 
30
+ This is a special release of Dolphin-DPO based on the LASER [paper](https://arxiv.org/pdf/2312.13558.pdf) and implementation by @fernandofernandes assisted by @ehartford
31
+
32
+ ```
33
+ @article{sharma2023truth,
34
+ title={The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction},
35
+ author={Sharma, Pratyusha and Ash, Jordan T and Misra, Dipendra},
36
+ journal={arXiv preprint arXiv:2312.13558},
37
+ year={2023} }
38
+ ```
39
+
40
+ We have further carried out a noise reduction technique based on SVD decomposition.
41
 
42
  We have adapted this paper on our own version of LASER, using Random Matrix Theory (Marchenko-Pastur theorem) to calculate optimal ranks instead of brute-force search.
43