make cache code work better with C

1. Check for malloc fails.  Return error if allocation fails.
2. in cache_size, avoid recalculating lenMod each loop.