Avoid illegal downcasts from BigPtrEntry to SwNode

...and while at it, remove the need for the TempBigPtrEntry derivation.

Change-Id: Ic758f646578a90a28632b1368fed209eb84f9c1c