Why not?  Just start with 

if (! fromServerHolder.getServer().equals(server.getServer()) ) {
  continue;
}

?