It's best to clean your Aeropress after each use to prevent any residual coffee grounds from affecting the flavor of your next cup.  However, if you're in a rush, it's okay to wait and clean it at the end of the day. Just make sure to thoroughly rinse all parts with hot water before storing.