SI-8134 SI-5954 Fix companions in package object under separate comp.

The tests cases enclosed exhibited two failures modes under
separate compilation.

  1. When a synthetic companion object for a case- or implicit-class
     defined in a package object is called for,
     `Namer#ensureCompanionObject` is used to check for an explicitly
     defined companion before decided to create a synthetic one.
     This lookup of an existing companion symbol by `companionObjectOf`
     would locate a symbol backed by a class file which was in the
     scope of the enclosing package class. Furthermore, because the
     owner of that symbol is the package object class that has now
     been noted as corresponding to a source file in the current
     run, the class-file backed module symbol is *also* deemed to
     be from the current run. (This logic is in `Run#compiles`.)

     Thinking the companion module already existed, no synthetic
     module was created, which would lead to a crash in extension
     methods, which needs to add methods to it.

  2. In cases when the code explicitly contains the companion pair,
     we still ran into problems in the backend whereby the class-file
     based and source-file based symbols for the module ended up in
     the same scope (of the package class). This tripped an assertion
     in `Symbol#companionModule`.

We get into these problems because of the eager manner in which
class-file based package object are opened in `openPackageModule`.
The members of the module are copied into the scope of the enclosing
package:

    scala> ScalaPackage.info.member(nme.List)
    res0: $r#59116.intp#45094.global#28436.Symbol#29451 = value List#2462

    scala> ScalaPackage.info.member(nme.PACKAGE).info.member(nme.List)
    res1: $r#59116.intp#45094.global#28436.Symbol#29451 = value List#2462

This seems to require a two-pronged defense:

  1. When we attach a pre-existing symbol for a package object symbol
     to the tree of its new source, unlink the "forwarder" symbols
     (its decls from the enclosing
     package class.

  2. In `Flatten`, in the spirit of `replaceSymbolInCurrentScope`,
     remove static member modules from the scope of the enclosing
     package object (aka `exitingFlatten(nestedModule.owner)`).

This commit also removes the warnings about defining companions
in package objects and converts those neg tests to pos (with
-Xfatal-warnings to prove they are warning free.)

Defining nested classes/objects in package objects still has
a drawback: you can't shift a class from the package to the
package object, or vice versa, in a binary compatible manner,
because of the `package$` prefix on the flattened name of
nested classes. For this reason, the `-Xlint` warning about
this remains. This issue is tracked as SI-4344.

However, if one heeds this warning and incrementatlly recompiles,
we no longer need to run into a DoubleDefinition error (which
was dressed up with a more specific diagnostic in SI-5760.)
The neg test case for that bug has been converted to a pos.
SI-5954 Invalidate TypeRef cache when opening package object

I noticed that the pos/5954d was tripping a println "assertion".
This stemmed from an inconsistency between
`TypeRef#{parents, baseTypeSeq}` for a package objects compiled
from source that also has a class file from a previous compilation
run.

I've elevated the println to a devWarning, and changed
`updatePosFlags`, the home of this evil symbol overwriting,
to invalidate the caches in the symbols info. Yuck.

I believe that this symptom is peculiar to package objects because
of the way that the completer for packages calls `parents` during
the Namer phase for package objects, before we switch the symbol
to represent the package-object-from-source. But it seems prudent
to defensively invalidate the caches for any symbol that finds its
way into `updatePosFlags`.
SI-5954 Add a TODO comment with an idea for future work

As the prophet once said:

  "'Tis better to never do at all than to have do and undo"

Let's try that in 2.12.
add -Xsource:version to scalac man page

The flag was added in d43618a (PR #3340) by @huitseeker.
SI-8134 Address pull request feedback

Aesthetics only.
SI-6736 Range.contains is wrong

Removed once-used private method that was calculating ranges in error and corrected the contains method (plus improved performance).
changes the order of whitebox typechecks. yes, again.

My first attempt at SI-6992 was about having whitebox expansions first
typecheck against outerPt and only then verify that the result is compatible
with innerPt.

That was a nice try, but soon after it went live in 2.11.0-M8, we've got
multiple reports with problems - both shapeless and then in a week specs2
started having issues with their whitebox macros.

In shapeless, typecheck against outerPt screwed up type inference, which
was more or less fixable by explicit type annotations, so I decided to
wait a bit before jumping to conclusions.

However, in specs2 the problem was more insidious. After being typechecked
against outerPt, expansions were being implicitly converted to a type
that became incompatible with innerPt. This revealed a fatal flaw of the
implemented approach - if allowed to typecheck against outerPt first,
whitebox macros could never be robust.

Now realizing that "outerPt > innerPt" doesn't work, I nevertheless wasn't
looking forward to rolling that back to "innerPt > outerPt", because that
would revive SI-6992 and SI-8048 that are highly unintuitive, especially
the latter one.

Therefore, this commit combines the permissiveness of "... > innerPt"
approaches with the robustness of "innerPt > outerPt", introducing
"WildcardType > innerPt > outerPt".
Reasonable Range operations consistently work when overfull.

Operations are reasonable when they don't require indexing or conversion into a collection.  These include head, tail, init, last, drop, take, dropWhile, takeWhile, dropRight, takeRight, span.

Tests added also to verify the new behavior.
SI-8226 Deduplicate Scala's Tokens and JavaTokens
SI-7475 Refactor findMember computation into a class

This affords us:

  - a better chance to share code with `findMembers` again
  - the ability to break up the logic into smaller methods for
    scrutability and JIT-tability.

The cost is the use of the heap rather than the stack for the
working area of the computation. But I didn't notice a difference
between `locker.lib` and `quick.lib` after this change.

I've left the old implementation in place with an assertion to
verify identical results. In the next commit, this stepping stone
will be removed, and we'll update `findMembers` to share the code.

Some problem areas have been highlighted and will be addressed
after the refactoring phase is complete.
SI-7475 findMember and findMembers: estranged no more

Swathes of important logic are duplicated between `findMember`
and `findMembers` after they separated on grounds of irreconcilable
differences about how fast they should run:

    d905558 Variation #10 to optimze findMember
    fcb0c01 Attempt #9 to opimize findMember.
    71d2ceb Attempt #8 to opimize findMember.
    77e5692 Attempty #7 to optimize findMember
    275115e Fixing problem that caused fingerprints to fail in
    e94252e Attemmpt #6 to optimize findMember
    73e61b8 Attempt #5 to optimize findMember.
    04f0b65 Attempt #4 to optimize findMember
    0e3c70f Attempt #3 to optimize findMember
    41f4497 Attempt #2 to optimize findMember
    1a73aa0 Attempt #1 to optimize findMember

This didn't actually bear fruit, and the intervening years have
seen the implementations drift.

Now is the time to reunite them under the banner of `FindMemberBase`.

Each has a separate subclass to customise the behaviour. This is
primarily used by `findMember` to cache member types and to assemble
the resulting list of symbols in an low-allocation manner.

While there I have introduced some polymorphic calls, the call sites
are only bi-morphic, and our typical pattern of compilation involves
far more `findMember` calls, so I expect that JIT will keep the
virtual call cost to an absolute minimum.

Test results have been updated now that `findMembers` correctly
excludes constructors and doesn't inherit privates.

Coming up next: we can actually fix SI-7475!
SI-7475 Private members are not inheritable

It turns out `findMembers` has been a bit sloppy in recent
years and has returned private members from *anywhere* up the
base class sequence. Access checks usually pick up the slack
and eliminate the unwanted privates.

But, in concert with the "concrete beats abstract" rule in
`findMember`, the following mishap appeared:

    scala> :paste
    // Entering paste mode (ctrl-D to finish)

    trait T { def a: Int }
    trait B { private def a: Int = 0 }
    trait C extends T with B { a }

    // Exiting paste mode, now interpreting.

    <console>:9: error: method a in trait B cannot be accessed in C
           trait C extends T with B { a }
                                      ^

I noticed this when compiling Akka against JDK 8; a new private method
in the bowels of the JDK was enough to break the build!

It turns out that some finesse in needed to interpret SLS 5.2:

> The private modifier can be used with any definition or declaration
> in a template. They are not inherited by subclasses [...]

So, can we simply exclude privates from all but the first base class?
No, as that might be a refinement class! The following must be
allowed:

    trait A { private def foo = 0; trait T { self: A => this.foo } }

This commit:

  - tracks when the walk up the base class sequence passes the
    first non-refinement class, and excludes private members
  - ... except, if we are at a direct parent of a refinement
    class itself
  - Makes a corresponding change to OverridingPairs, to only consider
    private members if they are owned by the `base` Symbol under
    consideration. We don't need to deal with the subtleties of
    refinements there as that code is only used for bona-fide classes.
  - replaces use of `hasTransOwner` when considering whether a
    private[this] symbol is a member.

The last condition was not grounded in the spec at all. The change
is visible in cases like:

    // Old
    scala> trait A { private[this] val x = 0; class B extends A { this.x } }
    <console>:7: error: value x in trait A cannot be accessed in A.this.B
           trait A { private[this] val x = 0; class B extends A { this.x } }
                                                                       ^
    // New
    scala> trait A { private[this] val x = 0; class B extends A { this.x } }
    <console>:8: error: value x is not a member of A.this.B
           trait A { private[this] val x = 0; class B extends A { this.x } }
                                                                       ^
Furthermore, we no longer give a `private[this]` member a free pass
if it is sourced from the very first base class.

    trait Cake extends Slice {
      private[this] val bippy = ()
    }

    trait Slice { self: Cake =>
      bippy // BCS: Cake, Slice, AnyRef, Any
    }

The different handling between `private` and `private[this]`
still seems a bit dubious.

The spec says:

> An different form of qualification is private[this]. A member M
> marked with this modifier can be accessed only from within the
> object in which it is defined. That is, a selection p.M is only
> legal if the prefix is this or O.this, for some class O enclosing
> the reference. In addition, the restrictions for unqualified
> private apply.

This sounds like a question of access, not membership. If so, we
should admit `private[this]` members from parents of refined types
in `FindMember`.

AFAICT, not too much rests on the distinction: do we get a
"no such member", or "member foo inaccessible" error? I welcome
scrutinee of the checkfile of `neg/t7475f.scala` to help put
this last piece into the puzzle.

One more thing: findMember does not have *any* code the corresponds
to the last sentence of:

> SLS 5.2 The modifier can be qualified with an identifier C
> (e.g. private[C]) that must denote a class or package enclosing
> the definition. Members labeled with such a modifier are accessible
> respectively only from code inside the package C or only from code
> inside the class C and its companion module (§5.4).
> Such members are also inherited only from templates inside C.

When I showed Martin this, he suggested it was an error in the
spec, and we should leave the access checking to callers of
that inherited qualified-private member.
SI-7475 Address review comments in FindMembers

  - comment the cases in `isNewMember`
  - check both sides for privacy (doesn't influence behaviour of
    any tests, but it means that method can be understood without
    understanding the order of traversal of members.)
  - hoist `findAll`
  - break out of the loop in `FindMembers` as soon as we determine
    that the candidate member is matched by an already-found member
    and can be discarded.
SI-8167 readLine shold flush output before reading input

As reported on scala-user:

> Using `scala.Predef.readLine(text: String, args: Any*)`
> on Windows causes strange behavior. If I am leaving some
> part of `text` unforced to be flushed (say, "---\nEnter new
>  expression >") then "Enter new expression >" flushed only
> after Enter press. Still, it works fine on Ubuntu and (seems like)
> on MacOS.
>
> My workaround is to force `java.lang.System.out` (that readLine
> depends on to write welcome string) to flush output like
> "---\nEnter new expression >\n".
SI-5994 battling implicits for String.lines

Deprecated ProcessBuilder's lines method and renamed it lineStream.

stream was another possibility, but that seemed more likely to cause conflicts.

streaming was also tried, but feedback was in favor of lineStream.

Also updated tutorial in ProcessBuilder.

I am sure this will break some tests, but because of the name conflict it's hard to be sure where they are.  So Jenkins gets to find the problems for me.

Changed name to lineStream.
Merge pull request #3504 from gkossakowski/readline-flush

 readLine shold flush output before reading input
Merge pull request #3456 from soc/SI-8226

SI-8226 Deduplicate JavaTokens/ScalaTokens
Tweak parser entry point for pq

Previously pq used pattern1 which required parens to be used in
alternative pattern. This commit tweaks it to allow pq"a | b"
syntax. Also adds some tests for alternative syntax.
Merge pull request #3505 from densh/topic/pq-entry

Tweak parser entry point for pq""
Merge pull request #3495 from xeno-by/ticket/8209

changes the order of whitebox typechecks. yes, again.