Remove unnecessary invalid downcasts

Change-Id: I04a6fa07ef4dfd2609b466c7c8ff9ba925c9683c