Cache in itself will not slow down any phone. But, if your phone memory is too low then having cache files won't help the cause and it is better to delete them. The same applies for Instagram cache too.