Note Proposes a way to fine-tune LLMs without actually fine-tuning the LLM itself. Instead uses trainable rank decomposition matrices with a fraction of parameters to fine-tune downstream tasks. Main idea: Weight updates in LMs during adaptation to a new task have a low intrinsic rank. Benefits: faster to train the much smaller matrices wrt. the LLM, saves memory, can have multiple such small matrices for individual downstream tasks (again saves memory).