The 29-year-old midfielder was made club captain following his move from Port Vale in 2014 and scored six goals in 70 appearances for the Gills.
But Loft, who won promotion from League Two in 2013 with Vale, was released by the Gills at the end of the season.
He becomes the second ex-Gillingham man to join the U's this summer, following the arrival of Brennan Dickenson.
Find all the latest football transfers on our dedicated page.
League Two side Colchester United have signed former Gillingham captain Doug Loft on a two-year deal.