Ignore .sass-cache files on deployment.
Remove executable bit from all asset permissions.
Stop deploying undesirable things: Gemfile, shell scripts, etc.
Delete removed files upon deployment.

Note that; however, if you exclude a file, it will not believed from S3 upon your next deployment.