Only add to unload queue if loaded.

Signed-off-by: Ross Allan <rallanpcl@gmail.com>