srowen commited on
Commit
1c75ca1
1 Parent(s): 0ea894a

Set "use_cache": true for faster generation

Browse files

We found that on the related Dolly v2 model, use_cache got set to false during training, but it really should be true for faster generation. Worked well! Could be applied to all of these similar models.
Ex: https://huggingface.co/databricks/dolly-v2-12b/commit/a7077365ca9caa324d6fdda760e953f2f75fac54

Files changed (1) hide show
  1. config.json +1 -1
config.json CHANGED
@@ -41,6 +41,6 @@
41
  },
42
  "torch_dtype": "float16",
43
  "transformers_version": "4.25.1",
44
- "use_cache": false,
45
  "vocab_size": 50260
46
  }
 
41
  },
42
  "torch_dtype": "float16",
43
  "transformers_version": "4.25.1",
44
+ "use_cache": true,
45
  "vocab_size": 50260
46
  }