scala/scala — pull request #4060 by mpociecha (2014-10-17 11:58:07+00:00)
Title: New classpath and sourcepath implementation

New classpath and sourcepath implementation

It's finished work started and abandoned by @gkossakowski
Related discussion: https://groups.google.com/forum/#!msg/scala-internals/xgb5YbKvQcw/7bcbsvKkzQkJ

The main goal of Grzegorz was to improve the efficiency of classpath. At the end it occurred that the efficiency improvement is not significant so Grzegorz stopped working on that.

On the other hand it was worth to continue these works due to several reasons:
- it's possible to reduce memory consumption significantly
- each possible efficiency improvement is important
- the old classpath is maybe too complex because it was created when e.g. .NET was supported - it can be simpler

The most of the code written by Grzegorz has been rewritten.

New (flat) classpath has dedicated classes for various file types characteristic for Scala with JVM backend - what allows us to create more efficient, more low-level operations.
To reduce memory consumption there's added caching for jar/zip files. So that we can reuse existing instances, when we create entries for the same files once again.

One of the most important things took into account during creating this implementation was adding it as an optional feature, which can be marked as experimental and turned on using flag. At the beginning users will use an old classpath implementation by default and they will be able to use the new implementation when:
- they will want to test it (feedback)
- they will want to gain benefits which it provides

I took into account also planned better support for JSR-223. So it supports for instance ManifestResources introduced here: https://github.com/scala/scala/pull/2238
There was no proper description what's going on with these changes and how to test them but @rjolly sent me some useful papers.
This particular type of classpath can be tested using e.g. command:
jrunscript -classpath scala-compiler.jar;scala-reflect.jar;scala-library.jar -l scala
where scala-library.jar was modified using https://github.com/rjolly/jarlister

I tested flat classpath in several ways:
- First of all obviously all tests (there are added certain new ones especially for this implementation) pass and bootstrapping also works.

- I built Scala IDE using my version of Scala and tested it with a very large codebase with about 100 projects. I turned off the closing of unused presentation compilers which I added lately. And after that I did certain custom operation which does a deep refresh of all projects (close projects, regenerate the workspace, open projects).

The flat classpath with turned on caching allowed me to reduce allocated memory in old gen from about 6GB (for current classpath) to about 750MB. So in the case of IDE and presentation compilers the gain is really significant.

- The efficiency of both implementations in real use cases seems to be similar but probably it could be tested better. In small benchmarks the flat one was winning.

- On the other hand there are situations when the gain is quite significant. When running a whole Scala's test suite:

Tests using a computer with SSD:

| Time type:| recursive classpath | flat classpath|
| ------------- |:-------------:| -----:|
| avg of 4 repetitions | 29 minutes 34 seconds | 27 minutes 15 seconds |
| min | 29 minutes 28 seconds | 27 minutes 6 seconds |
| max | 29 minutes 41 seconds | 27 minutes 32 seconds |

Tests using other computer with the scala project directory moved to some external HDD connected via USB cable:

| Time type:| recursive classpath | flat classpath|
| ------------- |:-------------:| -----:|
| avg of 4 repetitions | 56 minutes 43 seconds | 52 minutes 38 seconds |
| min | 55 minutes 40 seconds | 52 minutes 8 seconds |
| max | 58 minutes 26 seconds | 53 minutes 0 seconds |

Note: I changed the classPath method in Global to the dynamic dispatch. And now there's returned a base class of the old and the new classpath with common interface based on API of the old classpath. I tested it with sbt and there's no problem with compiling projects etc. Previously Grzegorz created certain hack for sbt's compiler interface where current classpath was calling the flat one, when it was needed. I changed this ugly solution to this mentioned dynamic dispatch and removed a hack. We can return to something like this, if it'd turn out that it's necessary.

Note2: FYI, commits of Grzegorz don't compile - as it was one big hack with some files which were not pushed. All my commits compile and for all of them JUnit tests pass. The bootstrapping and partest tests wouldn't work for some first ones when cp implementation type would be set to flat (firstly I had to make it working). But for several last ones they work without problems so finally I even commited a change where the flat classpath is set as a default one.

Anyway I squashed about 70 commits to just several ones trying by the way to preserve some way of thinking which is shown e.g. in the commit messages.

Lastly, the below disclaimer is required by the lawyers:

THIS PROGRAM IS SUBJECT TO THE TERMS OF THE BSD 3-CLAUSE LICENSE.

THE FOLLOWING DISCLAIMER APPLIES TO ALL SOFTWARE CODE AND OTHER MATERIALS CONTRIBUTED IN CONNECTION WITH THIS SOFTWARE:
THIS SOFTWARE IS LICENSED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE AND ANY WARRANTY OF NON-INFRINGEMENT, ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. THIS SOFTWARE MAY BE REDISTRIBUTED TO OTHERS ONLY BY EFFECTIVELY USING THIS OR
ANOTHER EQUIVALENT DISCLAIMER IN ADDITION TO ANY OTHER REQUIRED LICENSE TERMS.
ONLY THE SOFTWARE CODE AND OTHER MATERIALS CONTRIBUTED IN CONNECTION WITH THIS SOFTWARE, IF ANY, THAT ARE ATTACHED TO (OR OTHERWISE ACCOMPANY) THIS SUBMISSION (AND ORDINARY
COURSE CONTRIBUTIONS OF FUTURES PATCHES THERETO) ARE TO BE CONSIDERED A CONTRIBUTION. NO OTHER SOFTWARE CODE OR MATERIALS ARE A CONTRIBUTION.