Token Averaging A comprehensive research framework for analyzing whether averaging adjacent tokens in a LLM can reduce the compute compared to standard model. FAIRC/token-averaging-matched_mean-avg_50m_k8 Updated 7 days ago • 12 FAIRC/token-averaging-model1_50m Updated 7 days ago • 26 FAIRC/token-averaging-avg_50m_k8_wexp Updated 7 days ago • 12 FAIRC/token-averaging-avg_50m_k8_learnable_pos Updated 7 days ago • 8
MoWE Paper Releases associated with the MoWE paper written by Tavish, Vardhaman, Keshava, Dr. Deepan. Coded and trained with openlanguagemodel. FAIRC/mowe-mowe-768-19l 0.3B • Updated 8 days ago • 7 FAIRC/mowe-dense-768-19l 0.1B • Updated 8 days ago • 11 FAIRC/mowe-moe-768-19l 0.4B • Updated 8 days ago • 10
Token Averaging A comprehensive research framework for analyzing whether averaging adjacent tokens in a LLM can reduce the compute compared to standard model. FAIRC/token-averaging-matched_mean-avg_50m_k8 Updated 7 days ago • 12 FAIRC/token-averaging-model1_50m Updated 7 days ago • 26 FAIRC/token-averaging-avg_50m_k8_wexp Updated 7 days ago • 12 FAIRC/token-averaging-avg_50m_k8_learnable_pos Updated 7 days ago • 8
MoWE Paper Releases associated with the MoWE paper written by Tavish, Vardhaman, Keshava, Dr. Deepan. Coded and trained with openlanguagemodel. FAIRC/mowe-mowe-768-19l 0.3B • Updated 8 days ago • 7 FAIRC/mowe-dense-768-19l 0.1B • Updated 8 days ago • 11 FAIRC/mowe-moe-768-19l 0.4B • Updated 8 days ago • 10
