Check all requested partitions to make sure at least one is valid on job
submission.
Missed goto for last patch
Revert commit 3d69c6359a0cb64cfd6eb0661cba618d70984c50

While this will validate job at submit time, it results in redundant
looping when scheduling jobs. Working on alternate patch now.
Fix for job request with multiple partitions and node features

Without this patch, if the first listed partition lacks nodes
with required features the job would be rejected.
Remove commit 0ca48b70defb5d8341b361e7f3df6a9a778757e3

The original work this was based upon has been replaced with new
logic.
In sacctmgr, purge local qos list after update
Fix for scheduling batch jobs in multiple partitions

Without this change a high priority batch job may not start at submit
time. In addtion, a pending job with mutltiple partitions be cancelled
when the scheduler runs if any of it's partitions can not be used by
the job.
Add links to LBL Node Health Check program in FAQ and Download web pages
Improve example of SallocDefaultCommand in slurm.conf man page
Terminate sacctmgr on stdin EOF
Clarify use of job_submit/lua plugin
Merge branch 'slurm-2.5'

Conflicts:
	src/slurmctld/proc_req.c
Add support for configurable keep alive time for srun/slurmstep communications

Added "KeepAliveTime" configuration parameter

From Matthieu Hautreux:
TCP_KEEPALIVE addition. No TCP_KEEPALIVE seems to be configured in SLURM TCP
exchanges, thus letting the system potentially deadlocked if a remote host
dissapear and the local host is waiting on a read (the write would result in a
EPIPE or SIGPIPE depending on the masked signals). Adding keepalive with a
relatively large timeout value (5 minutes), could enhance the resilience of
SLURM for unexpected packet/connection loss without too much implication on the
scalability of the solution. The timeout could be configurable in case it is
find too aggresive for particular configurations.
Use shutdown() rather than close() for slurmstepd/srun sockets

From Matthieu Hautreux:
However, after discussing the point with onsite Bull support team and looking
at the slurmstepd code concerning stdout/err/in redirection we would like to
recommend two things for future versions of SLURM :

- sutdown(...,SHUT_WR) should be performed when managing the TCP sockets : no
shutdown(...,SHUT_WR) is performed on the TCP socket in slurmstepd eio
management. Thus, the close() can not reliably inform the other end of the
socket that the transmission is done (no TCP_FIN transmitted). As the close is
followed by an exit(), the kernel is the only entity that is knowing of the
fact that the close may not have been took into account by the other side (wich
might be our initial problem) and thus no retry can be performed, letting the
server side of the socket (srun) in a position where it can wait for a read
until the end of time.

- TCP_KEEPALIVE addition. No TCP_KEEPALIVE seems to be configured in SLURM TCP
exchanges, thus letting the system potentially deadlocked if a remote host
dissapear and the local host is waiting on a read (the write would result in a
EPIPE or SIGPIPE depending on the masked signals). Adding keepalive with a
relatively large timeout value (5 minutes), could enhance the resilience of
SLURM for unexpected packet/connection loss without too much implication on the
scalability of the solution. The timeout could be configurable in case it is
find too aggresive for particular configurations.
sacctmgr: terminate sacctmgr on EOF if readline function missing
Fix typo in comment
Terminate sreport on EOF
Merge branch 'slurm-2.5'

Conflicts:
	src/sacctmgr/sacctmgr.c
	src/sreport/sreport.c
Replace socket shutdown call with linger sockopt

The shutdown call was causing all pending I/O to be discarded.
Linger waits for pending I/O to complete before the close call returns.
Permit job with invalid QOS to run if QOS set by administrator

About a year ago I submitted a modification that you incorporated
into SLURM 2.4, which was to allow an admin to modify a job to use
a QOS even though the user did not have access to the QOS.

However, I must have tested it without having the Accounting set
to enforce QOS's. So, if an admin modifies a job to a QOS they
don't have access to, it will be modified, but the job will result
in a state of InvalidQOS, which is reasonable, since this would
handle the case where a user has their QOS removed. A problem,
however, is that even though the scheduler won't schedule the job,
backfill still will.

One approach would be to fix backfill to be consistent with
the scheduler (which should probably occur regardless), but
my thought would be to modify the scheduler to allow the QOS
as long as it was set by an admin, since that was the intent
of the modification to begin with.

I believe it  would only take a single line to change, just
adding a check on the job_ptr->limit_set_qos, to make sure
it was set by an admin:

                if (job_ptr->qos_id) {
                        slurmdb_association_rec_t *assoc_ptr;
                        assoc_ptr = (slurmdb_association_rec_t *)job_ptr->assoc_ptr;
                        if (assoc_ptr &&
                            !bit_test(assoc_ptr->usage->valid_qos,
                                      job_ptr->qos_id) &&
                            !job_ptr->limit_set_qos) {
                                info("sched: JobId=%u has invalid QOS",
                                        job_ptr->job_id);
                                xfree(job_ptr->state_desc);
                                job_ptr->state_reason = FAIL_QOS;
                                continue;
                        } else if (job_ptr->state_reason == FAIL_QOS) {
                                xfree(job_ptr->state_desc);
                                job_ptr->state_reason = WAIT_NO_REASON;
                        }
                }

Phil