We probably need to hold the schema lock here? In fact we probably want to hold the schema lock for the whole compact operation. That causes problems if we want to compact LSM trees though.