Constify arguments for locking KPIs where possible.

This enables locking consumers to pass their own structures around as const and
be able to assert locks embedded into those structures.

Reviewed by:	ed, kib, jhb

(cherry picked from commit a3e664d830a8cf18b39de7e24de3b6945e09d802)

Conflicts:
	share/man/man9/rwlock.9
Core structure and functions to support a feed-forward clock within the kernel.
Implement ffcounter, a monotonically increasing cumulative counter on top of the
active timecounter. Provide low-level functions to read the ffcounter and
convert it to absolute time or a time interval in seconds using the current
ffclock estimates, which track the drift of the oscillator. Add a ring of
fftimehands to track passing of time on each kernel tick and pick up updates of
ffclock estimates.

Committed on behalf of Julien Ridoux and Darryl Veitch from the University of
Melbourne, Australia, as part of the FreeBSD Foundation funded "Feed-Forward
Clock Synchronization Algorithms" project.

For more information, see http://www.synclab.org/radclock/

Submitted by:	Julien Ridoux (jridoux at unimelb edu au)

(cherry picked from commit 751092ac03de2a27db50a384379d53f52133c479)
Introduce macro stubs in the mutex implementation that will be always
defined and will allow consumers, willing to provide options, file and
line to locking requests, to not worry about options redefining the
interfaces.
This is typically useful when there is the need to build another
locking interface on top of the mutex one.

The introduced functions that consumers can use are:
- mtx_lock_flags_
- mtx_unlock_flags_
- mtx_lock_spin_flags_
- mtx_unlock_spin_flags_
- mtx_assert_
- thread_lock_flags_

Spare notes:
- Likely we can get rid of all the 'INVARIANTS' specification in the
  ppbus code by using the same macro as done in this patch (but this is
  left to the ppbus maintainer)
- all the other locking interfaces may require a similar cleanup, where
  the most notable case is sx which will allow a further cleanup of
  vm_map locking facilities
- The patch should be fully compatible with older branches, thus a MFC
  is previewed (infact it uses all the underlying mechanisms already
  present).

Comments review by:	eadler, Ben Kaduk
Discussed with:		kib, jhb
MFC after:	1 month

(cherry picked from commit 6a69e947d3eb5ffaceb04f999c1925ed771b7546)

Conflicts:
	sys/amd64/conf/GENERIC
	sys/vm/vm_map.c
Microoptimize time math.  As soon as our event periods are always below ome
second we may not add intereger parts by using bintime_addx() instead of
bintime_add().  Profiling shows handleevents() time redction by 15%.

(cherry picked from commit f443afa84822563372047962581a5565546be270)
Microoptimize LAPIC timer routines to avoid reading from hardware during
programming using earlier cached values. This makes respective routines to
disappear from PMC top and reduces total number of active CPU cycles on idle
24-core system by 10%.

(cherry picked from commit 8916b8f9033faa7d822c84a2bdd638979f11fab1)
Partialy MFcalloutng r236894 (by davide):
...
While here, Bruce Evans told me that "unsigned int" is spelled "u_int" in
KNF, so replace it where needed.

(cherry picked from commit 1fde1ad39cd741a3c648050cd16262f456975d73)
Particlly MFcalloutng r238425 (by davide):
Fix an issue related to old periodic timers. The code in kern_clocksource.c
uses interrupt to keep track of time, and this time may not match with
binuptime(). In order to address such incoherency, switch periodic timers
to binuptime().

Except further calloutng it is needed for already present cyclic subsystem.

(cherry picked from commit 6d37c6c863075ee08665b371b05f3cb68a371dd8)
Tweak the commit message in case of panic for sleeping from threads
with TDP_NOSLEEPING on.

The current message has no informations on the thread and wchan
involed, which may be useful in case where dumps have mangled dwarf
informations.

Reported by:    kib
Reviewed by:	bde, jhb, kib
MFC after:	1 week

(cherry picked from commit ac29dac65b70ec7f821f36ac07f086e6a514fe1a)
Improve check coverage about idle threads.

Idle threads are not allowed to acquire any lock but spinlocks.
Deny any attempt to do so by panicing at the locking operation
when INVARIANTS is on. Then, remove the check on blocking on a
turnstile.
The check in sleepqueues is left because they are not allowed to use
tsleep() either which could happen still.

Reviewed by:	bde, jhb, kib
MFC after:	1 week

(cherry picked from commit a634dfc320608f9955ab0350a7f78ee73064b909)

Conflicts:
	sys/kern/kern_mutex.c
	sys/kern/kern_rwlock.c
	sys/kern/kern_sx.c
panic() with reasonable message instead of returning zero frequency causing
division by zero later if event timer's minimal period is above one second.
For now it is just a theoretical possibility.

Found by:	Clang Static Analyzer

(cherry picked from commit 4a7b4c2440e0b30ec4b0f58c0a23fbe173f24e6b)
Pad tdq_lock to avoid false sharing with tdq_load and tdq_cpu_idle.

This enables CPU searches (which read tdq_load) to operate independently
of any contention on the spinlock.  Some scheduler-intensive workloads
running on an 8C single-socket SNB Xeon show considerable improvement with
this change (2-3% perf improvement, 5-6% decrease in CPU util).

Sponsored by:	Intel
Reviewed by:	jeff

(cherry picked from commit e56b4fdc17b3d9d906ac2a1a1fe90da82c774712)
Give mtx(9) the ability to crunch different type of structures, with the
only constraint that they have a lock cookie named mtx_lock.
This name, then, becames reserved from the struct that wants to use the
mtx(9) KPI and other locking primitives cannot reuse it for their
members.

Namely such structs are the current struct mtx and the new
struct mtx_padalign.  The new structure will define an object which is
the same as the same layout of a struct mtx but will be allocated in
areas aligned to the cache line size and will be as big as a cache line.

This is supposed to give higher performance for highly contented mutexes
both spin or sleep (because of the adaptive spinning), where the cache
line contention results in too much traffic on the system bus.

The struct mtx_padalign can be used in a completely transparent way
with the mtx(9) KPI.

At the moment, a possibility to MFC the patch should be carefully
evaluated because this patch breaks the low level KPI
(not its representation though).

Discussed with:	jhb
Reviewed by:	jeff, andre
Reviewed by:	mdf (earlier version)
Tested by:	jimharris

(cherry picked from commit b9e5ac8e1ce9f8dbec2f8ab42e0c36e0eb7c0244)

Conflicts:
	sys/kern/kern_mutex.c
Pad and align the callout_cpu mtx to its own cacheline to reduce false
sharing especially on the default CPU 0 callout_cpu structure.

This will be followed up by attilio@ with a conversion to the new struct
mtx_padalign but doing this manual conversion first gives an easy MFC
candidate since mtx_padalign is a more extensive system change.

Sponsored by:	Intel
Reviewed by:	jeff, attilio
MFC after:	1 week

(cherry picked from commit 8fbe05091504259bdb07f38b21e970c6f172b8cd)
Rework the known mutexes to benefit about staying on their own
cache line in order to avoid manual frobbing but using
struct mtx_padalign.

The sole exception being nvme and sxfge drivers, where the author
redefined CACHE_LINE_SIZE manually, so they need to be analyzed and
dealt with separately.

Reviwed by:	jimharris, alc

(cherry picked from commit d38d7bb24528def9112ecd75c967b4202e87cc4e)
replace bit shifting loop with 1<<fls(n), improve comments.

Reviewed by: davide

(cherry picked from commit d7137f4490527ab29539c6960e456c634dc6803b)
Get time of next event from other cores only if SMP is already started.

Reviewed by: mav
Obtained from: Semihalf

(cherry picked from commit d005d97a01379ab71eeddb9475961c5be0827934)
Fixup r246916 in case gcc is used to build.

Reported by:	attilio, simon

(cherry picked from commit da217eece1d0e73c39450e497b9dbd7564e1643a)
Introduce sbintime_t type -- the simplified version of struct bintime,
using 32.32 fixed point in form of single int64_t.  It is much easier to
use in cases where additional precision and range of struct bintime is
not required.

Reviewed by:	bde (previous version), davide

(cherry picked from commit f3c985cbe184c9d4c31d0e7140c146768a82833c)
MFcalloutng:
When CPU becomes idle, cpu_idleclock() calculates time to the next timer
event in order to reprogram hw timer. Return that time in sbintime_t to
the caller and pass it to acpi_cpu_idle(), where it can be used as one
more factor (quite precise) to extimate furter sleep time and choose
optimal sleep state. This is a preparatory change for further callout
improvements will be committed in the next days.

The commmit is not targeted for MFC.

(cherry picked from commit 2bf12d0c7ceced74a813d6940dd07310a11d6a2a)

Conflicts:
	sys/powerpc/powerpc/cpu.c
	sys/powerpc/wii/platform_wii.c
MFcalloutng (r247427 by mav):
We don't need any precision here. Let it be fast and dirty shift then
slow and excessively precise 64-bit division.

(cherry picked from commit 240074414d4919ebd889f27087c72783a7ed9b19)