Yes, you should remove damaged leaves from the rubber plant because it will improve the overall appearance of the plant, and keep it in good health. This can be done any time of the year to also encourage growth.