Package: memoria-pytorch
MemoriaMemoria is a general memory network that applies Hebbian theory which is a major theory explaining human memory formulation to enhance long-term dependencies in neural networks. Memoria stores and retrieves information called engram at multiple memory levels of working memory, short-term memory, and long-term memory, using connection weights that change according to Hebb's rule.Memoria is an independant module which can be applied to neural network models in various ways and the experiment code of the paper is in theexperimentdirectory.Please refer toMemoria: Hebbian Memory Architecture for Human-Like Sequential Processingfor more details about Memoria.Installation$pipinstallmemoria-pytorchYou can install memoria by pip command above.TutorialThis is a tutorial to help to understand the concept and mechanism of Memoria.1. Import Memoria and Set ParametersimporttorchfrommemoriaimportMemoria,EngramTypetorch.manual_seed(42)# Memoria Parametersnum_reminded_stm=4stm_capacity=16ltm_search_depth=5initial_lifespan=3num_final_ltms=4# Data Parametersbatch_size=2sequence_length=8hidden_dim=642. Initialize Memoria and Dummy DataFake random data and lifespan delta are used for simplification.memoria=Memoria(num_reminded_stm=num_reminded_stm,stm_capacity=stm_capacity,ltm_search_depth=ltm_search_depth,initial_lifespan=initial_lifespan,num_final_ltms=num_final_ltms,)data=torch.rand(batch_size,sequence_length,hidden_dim)3. Add Data as Working Memory# Add data as working memorymemoria.add_working_memory(data)# Expected values>>>len(memoria.engrams)16>>>memoria.engrams.data.shapetorch.Size([2,8,64])>>>memoria.engrams.lifespantensor([[3.,3.,3.,3.,3.,3.,3.,3.],[3.,3.,3.,3.,3.,3.,3.,3.]])4. Remind MemoriesEmpty memories are reminded because there is no engrams in STM/LTM yetreminded_memories,reminded_indices=memoria.remind()# No reminded memories because there is no STM/LTM engrams yet>>>reminded_memoriestensor([],size=(2,0,64))>>>reminded_indicestensor([],size=(2,0),dtype=torch.int64)5. Adjust Lifespan and MemoriesIn this step, no engrams earn lifespan because there is no reminded memoriesmemoria.adjust_lifespan_and_memories(reminded_indices,torch.zeros_like(reminded_indices))# Decreases lifespan for all engrams & working memories have changed into shortterm memory>>>memoria.engrams.lifespantensor([[2.,2.,2.,2.,2.,2.,2.,2.],[2.,2.,2.,2.,2.,2.,2.,2.]])>>>memoria.engrams.engrams_typestensor([[2,2,2,2,2,2,2,2],[2,2,2,2,2,2,2,2]],dtype=torch.uint8)>>>EngramType.SHORTTERM<EngramType.SHORTTERM:2>6. Repeat one more timeNow, there are some engrams in STM, remind and adjustment from STM will workdata2=torch.rand(batch_size,sequence_length,hidden_dim)memoria.add_working_memory(data2)>>>len(memoria.engrams)32>>>memoria.engrams.lifespantensor([[2.,2.,2.,2.,2.,2.,2.,2.,3.,3.,3.,3.,3.,3.,3.,3.],[2.,2.,2.,2.,2.,2.,2.,2.,3.,3.,3.,3.,3.,3.,3.,3.]])reminded_memories,reminded_indices=memoria.remind()# Remind memories from STM>>>reminded_memories.shapetorch.Size([2,6,64])>>>reminded_indices.shapetorch.Size([2,6])>>>reminded_indicestensor([[0,6,4,3,2,-1],[0,7,6,5,4,-1]])# Increase lifespan of all the reminded engrams by 5memoria.adjust_lifespan_and_memories(reminded_indices,torch.full_like(reminded_indices,5))# Reminded engrams got lifespan by 5, other engrams have got older>>>memoria.engrams.lifespan>>>memoria.engrams.lifespantensor([[6.,1.,6.,6.,6.,1.,6.,1.,2.,2.,2.,2.,2.,2.,2.,2.],[6.,1.,1.,1.,6.,6.,6.,6.,2.,2.,2.,2.,2.,2.,2.,2.]])7. RepeatRepeat 10 times to see the dynamics of LTM# This is default process to utilize Memoriafor_inrange(10):data=torch.rand(batch_size,sequence_length,hidden_dim)memoria.add_working_memory(data)reminded_memories,reminded_indices=memoria.remind()lifespan_delta=torch.randint_like(reminded_indices,0,6).float()memoria.adjust_lifespan_and_memories(reminded_indices,lifespan_delta)# After 10 iteration, some engrams have changed into longterm memory and got large lifespan# Engram type zero means those engrams are deleted>>>len(memoria.engrams)72>>>memoria.engrams.engrams_typestensor([[3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2],[0,0,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]],dtype=torch.uint8)>>>EngramType.LONGTERM<EngramType.LONGTERM:3>>>>EngramType.NULL<EngramType.NULL:0>>>>memoria.engrams.lifespantensor([[9.,1.,8.,2.,16.,5.,13.,7.,7.,3.,3.,4.,3.,3.,4.,2.,2.,1.,1.,1.,1.,1.,1.,1.,2.,6.,1.,1.,2.,2.,2.,2.,2.,2.,2.,2.],[-1.,-1.,3.,2.,19.,21.,11.,6.,14.,1.,5.,1.,5.,1.,5.,1.,1.,8.,2.,1.,1.,1.,2.,1.,1.,1.,1.,1.,2.,2.,2.,2.,2.,2.,2.,2.]])Citation@misc{park2023memoria,title={Memoria: Hebbian Memory Architecture for Human-Like Sequential Processing},author={Sangjun Park and JinYeong Bak},year={2023},eprint={2310.03052},archiveprefix={arXiv},primaryclass={cs.LG}}