Building matrix over time