Actually respect panic_on_empty config

Change-Id: I286a93211f645e1ee20af65326f3655ffa72f493