Remove unnecessary rows and cols.