GRIM: Fix deleting material when it is still needed



This fixes part of #1140. I had thought that the two issues in that issue were related. One, the changing colour, and two the save games breaking. This fixes the save games, but not the colour changing. It at least makes it easier to debug the colour changing.

I have not taken the time to go though the code to see if they affects anything else, and I think we don't generally call reference and dereference directly, but I figured I'd post this anyway.