Remove use of defaultdict

Remove the use of defaultdict to preserve Python 2.4 compatibility.