Only remove the canvas element from the container if it's currently attached.