Still guess an ID always, but only cache if it's a proper minion

Fix #8052
Fix #8050