This is likely due to a shortage of VRAM available on your system. Ray tracing can be memory hungry with more complex scenes. Try closing other 3D apps, and even browsers with a lot of tabs open, then retry your scene in Toolbag. You can also try reducing the resolution of your texture sets loaded in your scene and texture projects.