Simplify API for initially hooking custom-path providers into the planner.

Instead of register_custom_path_provider and a CreateCustomScanPath
callback, let's just provide a standard function hook in set_rel_pathlist.
This is more flexible than what was previously committed, is more like the
usual conventions for planner hooks, and requires less support code in the
core.  We had discussed this design (including centralizing the
set_cheapest() calls) back in March or so, so I'm not sure why it wasn't
done like this already.
Rearrange CustomScan API.

Make it work more like FDW plans do: instead of assuming that there are
expressions in a CustomScan plan node that the core code doesn't know
about, insist that all subexpressions that need planner attention be in
a "custom_exprs" list in the Plan representation.  (Of course, the
custom plugin can break the list apart again at executor initialization.)
This lets us revert the parts of the patch that exposed setrefs.c and
subselect.c processing to the outside world.

Also revert the GetSpecialCustomVar stuff in ruleutils.c; that concept
may work in future, but it's far from fully baked right now.
Rework echo_hidden for \sf and \ef from commit e4d2817.

PSQLexec's error reporting turns out to be too verbose for this case, so
revert to using PQexec instead with minimal error reporting. Prior to
calling PQexec, we call a function that mimics just the echo_hidden
piece of PSQLexec.
Fix mishandling of system columns in FDW queries.

postgres_fdw would send query conditions involving system columns to the
remote server, even though it makes no effort to ensure that system
columns other than CTID match what the remote side thinks.  tableoid,
in particular, probably won't match and might have some use in queries.
Hence, prevent sending conditions that include non-CTID system columns.

Also, create_foreignscan_plan neglected to check local restriction
conditions while determining whether to set fsSystemCol for a foreign
scan plan node.  This again would bollix the results for queries that
test a foreign table's tableoid.

Back-patch the first fix to 9.3 where postgres_fdw was introduced.
Back-patch the second to 9.2.  The code is probably broken in 9.1 as
well, but the patch doesn't apply cleanly there; given the weak state
of support for FDWs in 9.1, it doesn't seem worth fixing.

Etsuro Fujita, reviewed by Ashutosh Bapat, and somewhat modified by me
Allow simplification of EXISTS() subqueries containing LIMIT.

The locution "EXISTS(SELECT ... LIMIT 1)" seems to be rather common among
people who don't realize that the database already performs optimizations
equivalent to putting LIMIT 1 in the sub-select.  Unfortunately, this was
actually making things worse, because it prevented us from optimizing such
EXISTS clauses into semi or anti joins.  Teach simplify_EXISTS_query() to
suppress constant-positive LIMIT clauses.  That fixes the semi/anti-join
case, and may help marginally even for cases that have to be left as
sub-SELECTs.

Marti Raudsepp, reviewed by David Rowley
Detect PG_PRINTF_ATTRIBUTE automatically.

This eliminates gobs of "unrecognized format function type" warnings
under MinGW compilers predating GCC 4.4.
Fix memory leaks introduced by commit eca2b9b
Get rid of redundant production in plpgsql grammar.

There may once have been a reason for the intermediate proc_stmts
production in the plpgsql grammar, but it isn't doing anything useful
anymore, so let's collapse it into proc_sect.  Saves some code and
probably a small number of nanoseconds per statement list.

In passing, correctly alphabetize keyword lists to match pl_scanner.c;
note that for "rowtype" vs "row_count", pl_scanner.c must sort on the
basis of the lower-case spelling.

Noted while fooling with a patch to de-reserve more plpgsql keywords.
Distinguish XLOG_FPI records generated for hint-bit updates.

Add a new XLOG_FPI_FOR_HINT record type, and use that for full-page images
generated for hint bit updates, when checksums are enabled. The new record
type is replayed exactly the same as XLOG_FPI, but allows them to be tallied
separately e.g. in pg_xlogdump.
Add a few paragraphs to B-tree README explaining L&Y algorithm.

This gives an overview of what Lehman & Yao's paper is all about, so that
you can understand the rest of the README without having to read the paper.

Per discussion with Peter Geoghegan and others.
Make UPDATE privileges distinct from INSERT privileges in RTEs

Previously, relation range table entries used a single Bitmapset field
representing which columns required either UPDATE or INSERT privileges,
despite the fact that INSERT and UPDATE privileges are separately
cataloged, and may be independently held.  This worked because
ExecCheckRTEPerms() was called with a ACL_INSERT or ACL_UPDATE
requiredPerms, and based on that it was evident which type of
optimizable statement was under consideration.  Since historically no
type of optimizable statement could directly INSERT and UPDATE at the
same time, there was no ambiguity as to which privileges were required.

This largely mechanical commit is required infrastructure for the
INSERT...ON CONFLICT UPDATE feature, which introduces an optimizable
statement that may be subject to both INSERT and UPDATE permissions
enforcement.  Tests follow in a later commit.

Note that this commit necessitates an initdb, since stored ACLs are
broken.
Support INSERT ... ON CONFLICT {UPDATE | IGNORE}

This non-standard INSERT clause allows DML statement authors to specify
that in the event of each of any of the tuples being inserted
duplicating an existing tuple in terms of a value or set of values
constrained by an "arbitrating" unique index, an alternative path may be
taken.  The statement may alternatively IGNORE the tuple being inserted
without raising an error, or go to UPDATE the existing tuple whose value
is duplicated by a value within one single tuple proposed for insertion.
The implementation loops until either an insert or an UPDATE/IGNORE
occurs.  No existing tuple may be affected more than once per INSERT.

This is implemented using a new infrastructure called "speculative
insertion".  A speculative inserter establishes the right to insert
ahead of time.  In order to proceed with insertion, consensus must be
reached between all unique indexes on the table, or else the alternative
path is taken.  Finally, heap tuple insertion occurs in the conventional
manner, followed by insertion of index tuples, where index tuple
insertion picks up where value locking left off, performing essentially
the conventional insertion process in a staggered fashion.
Alternatively, we may go to UPDATE, using the EvalPlanQual() mechanism
to execute a special auxiliary plan.

READ COMMITTED isolation level is permitted to UPDATE a tuple even where
no version is visible to the command's MVCC snapshot.  Similarly, any
query predicate associated with the UPDATE portion of the new statement
need only satisfy an already locked, conclusively committed and visible
conflict tuple.  When the predicate isn't satisfied, the tuple is still
locked, which implies that at READ COMMITTED, a tuple may be locked
without any version being visible to the command's MVCC snapshot.

Users specify a single unique index to take the alternative path on,
which is inferred from a set of user-supplied column names (or
expressions).  This is mandatory for the ON CONFLICT UPDATE variant,
which should address concerns about spuriously taking an incorrect
alternative ON CONFLICT path (i.e. the wrong unique index is used for
arbitration of whether or not to take the alternative path) due to there
being more than one would-be unique violation.  Previous revisions of
the patch didn't mandate this.  However, we may still IGNORE based on
the first would-be unique violation detected, on the assumption that it
doesn't particularly matter where it originated from for that variant
(iff the user didn't make a point of indicated his or her intent).

The auxiliary ModifyTable plan used by the UPDATE portion of the new
statement is not formally a subplan of its parent INSERT ModifyTable
plan.  Rather, it's an independently planned subquery, whose execution
is tightly driven by its parent.  Special auxiliary state pertaining to
the auxiliary UPDATE is tracked by its parent through all stages of
query execution.

The implementation imposes some restrictions on child auxiliary UPDATE
plans, which make the plans comport with their parent to the extent
required during the executor stage.  One user-visible consequences of
this is that the special auxiliary UPDATE query cannot have subqueries
within its targetlist or WHERE clause.  UPDATEs may not reference any
other table, and UPDATE FROM is disallowed.  INSERT's RETURNING clause
continues to only project tuples actually inserted.
EXCLUDED expressions within ON CONFLICT UPDATE

EXCLUDED.* (which previously appeared as EXCLUDED(), and CONFLICTING()
before that) is an "internal" primnode expression which enables
referencing of rejected-for-insertion tuples within both the targetlist
and predicate of the UPDATE portion of an INSERT ... ON CONFLICT UPDATE
query.  The expression is invoked using an alias-like syntax (more on
how this works later).  The fact that a dedicated expression is used
(rather than a dedicated range table entry involved in query
optimization) is an implementation detail.

This additional support is particularly useful for ON CONFLICT queries
that propose multiple tuples for insertion, since it isn't otherwise
possible to succinctly decide which actual values to update each column
with (in the event of taking the update path in respect of a given
slot).

The effects of BEFORE INSERT row triggers on the slot/tuple proposed for
insertion are carried.  This seems logical, since it might be the case
that the rejected values would not have been rejected had some BEFORE
INSERT trigger been disabled.  On the other hand, the potential hazards
around equivalent modifications occurring when both INSERT and UPDATE
BEFORE triggers are fired for the same slot/tuple should be considered
by client applications.  It's possible to imagine a use case in which
this behavior is surprising and undesirable -- essentially the same
non-idempotent modification may occur twice.  (It might also be the case
that BEFORE trigger related side-effects undesirably occur twice, but
writing BEFORE triggers with external side-effects is already considered
a questionable practice for several reasons (consider commit 6868ed74),
and besides, the implementation cannot reasonably prevent this, as noted
in nodeModifyTable.c comments added by the main ON CONFLICT commit).

In this revision, the raw grammar does not generate an ExcludedExpr.
Parse analysis of ON CONFLICT UPDATE is made to add a new relation RTE
to the auxiliary sub_pstate parser state (an alias for the target).
This makes parse analysis build a query tree that is more or less
consistent with there actually being an EXCLUDED relation.  Then, as
part of query rewrite, immediately after normalizing the UPDATE
targetlist, Vars referencing the pseudo-relation (using the EXCLUDED
alias) are replaced with ExcludedExpr that references Vars in the target
relation itself.

Speculative insertion/the executor arranges to rig the Vars and
UPDATE-related/EPQ scan planstate's expression context such that values
will actually originate from the rejected tuple's slot (driven, as
always for the UPDATE's execution, by the parent INSERT ModifyTable
node, changed once per slot proposed for insertion as appropriate).
This whole mechanism is somewhat similar to the handling of trigger WHEN
clauses, where a similar dance must also occur within the executor.

Note that pg_stat_statements does not fingerprint ExludedExpr, because
it cannot appear in the post-parse-analysis, pre-rewrite Query tree.
(pg_stat_statements does not fingerprint every primnode anyway, mostly
because some are only expected in utility statements).  Other existing
Node handling sites that don't expect to see primnodes that appear only
after rewriting (ExcludedExpr may be in its own subcategory here in that
it is the only such non-utility related Node) do not have an
ExcludedExpr case added either.
Project updates from ON CONFLICT UPDATE RETURNING

This establishes that an INSERT with an ON CONFLICT UPDATE clause
processes all slots that are ultimately affected, regardless of whether
or not the alternative ON CONFLICT UPDATE path was taken.  However, if
an ON CONFLICT UPDATE's WHERE clause is not satisfied in respect of some
slot/tuple, the post-update tuple is not projected (although the row is
still locked, just as before).

Also, for ON CONFLICT UPDATE variant INSERTs (but not ON CONFLICT IGNORE
variant INSERTs), the number of rows affected in total is reported by
the command tag using the new "UPSERT" command identifier, which
otherwise matches the format of the existing "INSERT" command tag.

There is no precedent for a top level command that uses a different
command tag identifier according to whether or not some clause was used,
but doing so seems appropriate, since client programs are expected to
have an interest in whether or not some number of rows projected by
RETURNING may have been updated, and in any case indicating that the
rows were affected by an "INSERT" when they may not have been inserted
is simply misleading.  However, there is still no principled method for
client programs to distinguish between INSERT ...  ON CONFLICT UPDATE
projected tuples generated by being inserted or by being updated.  This
is thought not to matter, since the use of INSERT with ON CONFLICT
UPDATE indicates that either outcome is equivalent.
Tests for INSERT ... ON CONFLICT {UPDATE | IGNORE}

Add dedicated isolation tests for both UPDATE and IGNORE variants,
illustrating the "MVCC violation" that allows a READ COMMITTED
transaction's UPDATE to succeed in updating a tuple with no version
visible to its command's MVCC snapshot.  Add regression tests, which for
the most part are intended to exercise interactions with other features
(e.g.  updatable views, inheritance, triggers).

Add a few general purpose smoke tests too, testing everything from
EXPLAIN output to unique index inference.
Internal documentation for INSERT ... ON CONFLICT {UPDATE | IGNORE}

Includes documentation for both the nbtree and executor READMEs, as well
as the user-facing SGML documentation around index AMs.  The latter two
additions should make clear how the AM interface has been revised for
amcanunique AMs in the abstract, without reference to the only current
implementation, nbtree.
User-visible documentation for INSERT ... ON CONFLICT {UPDATE | IGNORE}

INSERT ... ON CONFLICT {UPDATE | IGNORE} is documented as a new clause
of the INSERT command.  Some potentially surprising interactions with
triggers are noted -- BEFORE INSERT per-row triggers must fire without
the INSERT path necessarily being taken, for example.

All the existing features that INSERT ... ON CONFLICT {UPDATE | IGNORE}
fails to completely play nice with have those limitations noted.  (Notes
are added to the existing documentation for those other features,
although some of these cases will need to be revisited).  This includes
postgres_fdw, updatable views and table inheritance (although these have
most interesting cases covered, particularly inheritance).

Finally, a user-level description of the new "MVCC violation" that the
ON CONFLICT UPDATE variant sometimes requires has been added to "Chapter
13 - Concurrency Control", beside existing commentary on READ COMMITTED
mode's special handling of concurrent updates.  The new "MVCC violation"
introduced seems somewhat distinct from the existing one (i.e.  what is
internally referred to as the EvalPlanQual() mechanism), because in READ
COMMITTED mode it is no longer necessary for any row version to be
conventionally visible to the command's MVCC snapshot for an UPDATE of
the row to occur (or for the row to be locked, should the WHERE clause
predicate not be satisfied).