You will know if your siding is rotten if, it is falling off, you are finding holes in the siding, if the corners of the siding comes off, if you find soft areas, if the paining is bubbling and if the painting color is changing.