Fix source removal logic; fold it in with patch detection