Program: 60848
Project: 60848
Buggy source:
/** 
 * Look at what is in meta and the list of servers that have checked in and make reconciliation. We cannot tell definitively the difference between a clean shutdown and a cluster that has been crashed down. At this stage of a Master startup, they look the same: they have the same state in hbase:meta. We could do detective work probing ZK and the FS for old WALs to split but SCP does this already so just let it do its job. <p>>The profiles of clean shutdown and cluster crash-down are the same because on clean shutdown currently, we do not update hbase:meta with region close state (In AMv2, region state is kept in hbse:meta). Usually the master runs all region transitions as of AMv2 but on cluster controlled shutdown, the RegionServers close all their regions only reporting the final change to the Master. Currently this report is ignored. Later we could take it and update as many regions as we can before hbase:meta goes down or have the master run the close of all regions out on the cluster but we may never be able to achieve the proper state on all regions (at least not w/o lots of painful manipulations and waiting) so clean shutdown might not be possible especially on big clusters.... And clean shutdown will take time. Given this current state of affairs, we just run ServerCrashProcedure in both cases. It will always do the right thing.
 * @return True if for sure this is a failover where a Master is starting up into an alreadyrunning cluster.
 */
private boolean processofflineServersWithOnlineRegions(){
  boolean deadServers=!master.getServerManager().getDeadServers().isEmpty();
  final Set<ServerName> offlineServersWithOnlineRegions=new HashSet<>();
  int size=regionStates.getRegionStateNodes().size();
  final List<RegionInfo> offlineRegionsToAssign=new ArrayList<>(size);
  long startTime=System.currentTimeMillis();
  boolean failover=deadServers;
  for (  RegionStateNode regionNode : regionStates.getRegionStateNodes()) {
    if (regionNode.getState() == State.OPEN) {
      final ServerName serverName=regionNode.getRegionLocation();
      if (!master.getServerManager().isServerOnline(serverName)) {
        offlineServersWithOnlineRegions.add(serverName);
      }
 else {
        failover=true;
      }
    }
 else     if (regionNode.getState() == State.OFFLINE) {
      if (isTableEnabled(regionNode.getTable())) {
        offlineRegionsToAssign.add(regionNode.getRegionInfo());
      }
    }
  }
  for (  ServerName serverName : offlineServersWithOnlineRegions) {
    if (!master.getServerManager().isServerOnline(serverName)) {
      LOG.info("String_Node_Str" + serverName + "String_Node_Str");
      killRegionServer(serverName);
    }
  }
  setFailoverCleanupDone(true);
  if (offlineRegionsToAssign.size() > 0) {
    master.getMasterProcedureExecutor().submitProcedures(master.getAssignmentManager().createRoundRobinAssignProcedures(offlineRegionsToAssign));
  }
  return failover;
}

Fixed version (ground truth):
/** 
 * Look at what is in meta and the list of servers that have checked in and make reconciliation. We cannot tell definitively the difference between a clean shutdown and a cluster that has been crashed down. At this stage of a Master startup, they look the same: they have the same state in hbase:meta. We could do detective work probing ZK and the FS for old WALs to split but SCP does this already so just let it do its job. <p>>The profiles of clean shutdown and cluster crash-down are the same because on clean shutdown currently, we do not update hbase:meta with region close state (In AMv2, region state is kept in hbse:meta). Usually the master runs all region transitions as of AMv2 but on cluster controlled shutdown, the RegionServers close all their regions only reporting the final change to the Master. Currently this report is ignored. Later we could take it and update as many regions as we can before hbase:meta goes down or have the master run the close of all regions out on the cluster but we may never be able to achieve the proper state on all regions (at least not w/o lots of painful manipulations and waiting) so clean shutdown might not be possible especially on big clusters.... And clean shutdown will take time. Given this current state of affairs, we just run ServerCrashProcedure in both cases. It will always do the right thing.
 * @return True if for sure this is a failover where a Master is starting up into an alreadyrunning cluster.
 */
private boolean processofflineServersWithOnlineRegions(){
  boolean deadServers=!master.getServerManager().getDeadServers().isEmpty();
  final Set<ServerName> offlineServersWithOnlineRegions=new HashSet<>();
  int size=regionStates.getRegionStateNodes().size();
  final List<RegionInfo> offlineRegionsToAssign=new ArrayList<>(size);
  boolean failover=deadServers;
  for (  RegionStateNode regionNode : regionStates.getRegionStateNodes()) {
    if (regionNode.getState() == State.OPEN) {
      final ServerName serverName=regionNode.getRegionLocation();
      if (!master.getServerManager().isServerOnline(serverName)) {
        offlineServersWithOnlineRegions.add(serverName);
      }
 else {
        failover=true;
      }
    }
 else     if (regionNode.getState() == State.OFFLINE) {
      if (isTableEnabled(regionNode.getTable())) {
        offlineRegionsToAssign.add(regionNode.getRegionInfo());
      }
    }
  }
  for (  ServerName serverName : offlineServersWithOnlineRegions) {
    if (!master.getServerManager().isServerOnline(serverName)) {
      LOG.info("String_Node_Str" + serverName + "String_Node_Str");
      killRegionServer(serverName);
    }
  }
  setFailoverCleanupDone(true);
  if (offlineRegionsToAssign.size() > 0) {
    master.getMasterProcedureExecutor().submitProcedures(master.getAssignmentManager().createRoundRobinAssignProcedures(offlineRegionsToAssign));
  }
  return failover;
}