Merge branch 'New3' of https://github.com/ayende/ravendb into New3
Squashed 'Raven.Voron/' changes from b6b6715..7ac0866

7ac0866 Merge branch 'voron' of https://github.com/ayende/raven.voron into voron
7ff4da2 adjustments to csv output, added the ability to switch it on and off
9936830 fixed issues with SqlLite tests
5ea7f62 cleanup
12b8915 There is no need to switch over immediately to the data file, and it may cause problems (the log file data is already cached, probably)
6ea13d5 Making sure that we surface errors in flushing as soon as possible (next write)
3d7d4a5 Fixing copying free space root to root tree entry
f43550f After flushing a log file, forget all about the earlier pages in the log file, use the ones directly from the data file
000e380 When we have a write transaction that didn't actually write anything, just ignore it from storage & tx id perspectives.
3fc5c28 Even if there isn't any usable data in a log file, we might still have some usable pages there.
4c4cdb8 Make sure that we start writing from the last written position after restart
7b63ffe Optimizing the test
70e425c Making sure NOT to use page information from corrupted transactions
2386613 Using ZipArchive and not Package (no need for WindowsBase dependency)
9857a48 Merge branch 'voron' of github.com:ayende/raven.voron into voron
185d3ef Merge branch 'voron' of https://github.com/ayende/raven.voron into voron
f06581c Better test
ff96330 Fixing Incremental Backup
b5fe8b0 Fixing typo
2e208ea Fixing disposal of journal files. Fixing deletion of fully synced journals
7aab724 lmdb - initial read parallel
54ebf1c lmdb perf test - better code
e52e6d6 parallel write for esent
de91e56 write parallel for lmdb
65edcf8 cleanup in write parallel processing
515cc8a Merge branch 'voron' of https://github.com/ayende/raven.voron into voron
327d420 Fixing issues that popped out after the move to scratch mode Mostly with regards to completely full journals
9429e72 lmdb test now works properly
33b6e2b parallel writes for sql ce
9205651 parallel writes for sql lite
d575475 added write parallel for sql server
a95b031 added parallel writes to voron
ad1f200 Lmdb perf test - makefile (separated from the lmdb source code)
31afabb Merge branch 'voron' of github.com:ayende/raven.voron into voron
262ace6 Test perf

git-subtree-dir: Raven.Voron
git-subtree-split: 7ac0866af8e21b2f386a02286b468c4b1cc38064
Merge commit '8c2ca041f9f4813dae83e3c0e5a86cee0cc36153' into New3
all conflicts resolved 'using theirs'
Conflicts:
	Raven.Voron/Performance.Comparison/Performance.Comparison/Esent/EsentTest.cs
	Raven.Voron/Performance.Comparison/Performance.Comparison/IStoragePerformanceTest.cs
	Raven.Voron/Performance.Comparison/Performance.Comparison/LMDB/LmdbTest.cs
	Raven.Voron/Performance.Comparison/Performance.Comparison/Performance.Comparison.csproj
	Raven.Voron/Performance.Comparison/Performance.Comparison/Program.cs
	Raven.Voron/Performance.Comparison/Performance.Comparison/SQLCE/SqlCeTest.cs
	Raven.Voron/Performance.Comparison/Performance.Comparison/SQLServer/SqlServerTest.cs
	Raven.Voron/Performance.Comparison/Performance.Comparison/SQLite/SqlLiteTest.cs
	Raven.Voron/Performance.Comparison/Performance.Comparison/StoragePerformanceTestBase.cs
	Raven.Voron/Performance.Comparison/Performance.Comparison/Voron/VoronTest.cs
	Raven.Voron/Voron.Tests/Backups/Full.cs
	Raven.Voron/Voron.Tests/Backups/Incremental.cs
	Raven.Voron/Voron.Tests/Bugs/RecoveryMultipleJournals.cs
	Raven.Voron/Voron.Tests/Journal/ApplyingPolicies.cs
	Raven.Voron/Voron.Tests/Storage/BigValue.cs
	Raven.Voron/Voron.Tryout/Program.cs
	Raven.Voron/Voron/Impl/Backup/FullBackup.cs
	Raven.Voron/Voron/Impl/Backup/IncrementalBackup.cs
	Raven.Voron/Voron/Impl/Journal/JournalFile.cs
	Raven.Voron/Voron/Impl/Journal/JournalReader.cs
	Raven.Voron/Voron/Impl/Journal/WriteAheadJournal.cs
	Raven.Voron/Voron/Impl/Paging/FilePager.cs
	Raven.Voron/Voron/Impl/Transaction.cs
	Raven.Voron/Voron/StorageEnvironment.cs
	Raven.Voron/Voron/StorageEnvironmentOptions.cs
	Raven.Voron/Voron/Voron.csproj
minor fixes after Voron subtree pull
git glitch --> no changes were made to the file
crlf + build fixes
RavenDB-1379 - StartsWith should not count excluded documents
RavenDB-1305. Bulk insert will throw concurrency exception on duplicate keys
minor
Merge branch 'RavenDB-1305' of https://github.com/ppekrol/ravendb into New3
Merge branch 'RavenDB-1379' of https://github.com/ppekrol/ravendb into New3
RavenDB-851. Bug with Count() > x in Where()?
Merge pull request #484 from ppekrol/RavenDB-851

RavenDB-851. New exception text
Merge branch 'New3' of https://github.com/myarichuk/ravendb into New3

Conflicts:
	Raven.Voron/Performance.Comparison/Other/LMDB/Makefile
	Raven.Voron/Performance.Comparison/Other/LMDB/lmdb_perf_test.cpp
	Raven.Voron/Performance.Comparison/Performance.Comparison/PerfTracker.cs
Merge branch 'New3' of github.com:ayende/ravendb into New3

Conflicts:
	Raven.Voron/Performance.Comparison/Other/LMDB/Makefile
	Raven.Voron/Performance.Comparison/Other/LMDB/lmdb_perf_test.cpp
	Raven.Voron/Performance.Comparison/Performance.Comparison/PerfTracker.cs
Squashed 'Raven.Voron/' changes from 7ac0866..538de07

538de07 Making sure that we register and wait for all the async disposal tasks
8917576 making sure that scratch file is deleted
4129625 build fix
9d6157a async release only for scratch pager
83f8b9f removed lock from get snapshot in JournalFile completely
f345779 nothing relevant - fixing doc headers
593534a executing accessor disposal in task to not block the running thread
9a41b4e Merge branch 'voron' of github.com:ayende/raven.voron into voron
20b62d7 RavenDB-1488 Voron: Reduce memory move cost during updates - implementation for overflows
00fb989 removed unnecessary ReadWrite tx in journal applicator
0e365ef decreased lock scope in JournalFile write
4b7dc63 Don't lock unless you have good reason to
b7ed01c Moving away completely from immutable collections
8473763 Adding todos
adf0866 Moving from ImmutableDictionary to SafeDictionary
4ecafed Fixing bad line placement
d1d1e7f Introducing Safelist instead of ImmutableList Will cause a new instance (via copying) when we call it. More expensive for GC, but much faster for CPU in general
9270586 Removing immutable list from write ahead journal
1f11bdd Removing mutable collections from pure memory journals
b65ce75 minor
2c4cd34 Check for errors in SetFileLength native method
a89d0e6 reinitializing transaction end positions after restart
56dd96b fix for last synced page calculation
e7bf145 Tests that exposes the issue related with invalid usage of page translation table to determine the last synced page number
ad493bf Reproducing error consistently
7933882 Minor refactoring
6866816 Merge branch 'voron' of github.com:ayende/raven.voron into voron
a16af36 Better handling of recovery event
09c1d85 Fixing tests that intentionally corrupt pages
4d826d6 Fix against ObjectDisposedException: The semaphore has been disposed.
580bc8a Fixing against 'collection was modified' exception
811a39e RavenDB-1281 Quotas implementation
bd2f82d Fixing potential mutable read & access issue
b31bb0c Revert "RavenDB-1281 Voron: Implement quotas"
54fbd4e validating _lastSyncedPage calculation during debug in journal applicator
394f22d Merge branch 'voron' of https://github.com/ayende/raven.voron into voron
b48382b RavenDB-1281 Voron: Implement quotas
1671ae8 Merge branch 'voron' of https://github.com/ayende/raven.voron into voron
49aecd8 added OnRecoveryError even handler to storage options
0cd4cbe Removing no longer valid environment stat
e93cffc Merge branch 'voron' of https://github.com/ayende/raven.voron into voron
3445edb removed setItems from JournalFile on immutable dict - performance bottleneck
aaf9034 RavenDB-1486 Voron: ensure free pages on tx rollback/dispose
73adfc9 fixing tests
00c81a4 fixing issue with lastSyncedPage calculation for overflows
4f72be3 more fixes in lastProcessedJournal calculation
21a0003 Fixing lastProcessedJournalPage calculation
3d5e8bd Missing item
ef90159 added failing recovery test for large values
a95a52e added find graphviz method
3659da1 Properly move the pointer that indicates the last used page in the scratch buffer pool (in order to avoid overwrites of scratch pages by overflows allocation)
44ffd7b Fixing the build
5a0bda9 Avoid allocating Source info all the time
0d32ce9 - Smaller # of transaction to get to error faster (still cause issue) - Have a proper way to stop early on error and redo the issue.
c4ef2fc - Adding the source for each page. - Fixing namespaces
853d61a We _have_ to lock on disposal of WriteAheadJournal
cbd15cb Minor
9461599 Fixing the merge, we already already doing locking in journal applicator now.
5131597 Merge branch 'voron' of github.com:ayende/raven.voron into voron
4645375 Flush to data file should have own scratch pager reference
0863747 removed recently added wait in WriteAheadJournal disposal - maybe it should be there?
d73c56d fixing issue with existing database recovery - assigning proper journal file number
0c6a4d2 all apps need to be run as administrator. Added app.manifest
058296e fixing issue in journal applicator with determining when journal is used or not
1330f17 Added failing test
1c851e6 Merge branch 'voron' of github.com:ayende/raven.voron into voron
6f3be5b Use built-in methods of immutable collections to do updates (because FreeScratchPagesOlderThan method can be executed concurrently and assigned references can change in the meanwhile)
5ccb3f0 - added locker to be sure that ApplyLogsToDataFile will not be executed more than once at the same time - removed unused method in StorageEnvironment
ed02bed passing tx to ensure continuous in IncrementalBackup
b621368 Ensure that when we use a log snapshot's page translation table to read from the scratch we protect ourself against read the overwritten scratch page (if page is already written to the data file then read from there)
86b5179 There is no need to add reference to the journal file by transaction since it no longer reads directly from it (the intention was to add a reference because read transactions read from them by using journal snapshots)
4c1022c Added failing tests that exposes the issue related to concurrent reading and flushing
01579cc Merge branch 'voron' of github.com:ayende/raven.voron into voron
c66b201 Debug stuff
d5ed90e Removing unused code
5a1da94 Merge branch 'voron' of github.com:ayende/raven.voron into voron
d4a52b0 Making sure that there can only be a single flushing operation at one time
28d8f3d Fixing the calculation of _lastSyncedJournal and _lastSyncedPage
9a40afc Better exception message
a1de505 When you get unused journals use snapshots not the files directly
7a44bdc Making sure that we are using sync commit
c8e2196 Fixing connection string
67acbc5 Merge branch 'voron' of github.com:ayende/raven.voron into voron
8b77783 Merge branch 'voron' of https://github.com/ayende/raven.voron into voron
f24db58 fixed issue with release PagerState during async writing to data file, which was causing exceptions in other write transactions
7207a55 Merge branch 'voron' of https://github.com/jahmai/raven.voron into voron
1a9ad96 * Read transaction header as long as it really is a header (by checking TransactionHeaderMarker)
1ee34ed Added failing test
a739602 assigning tx id after lock
cd6fa17 precaution during storage env disposal
6436bdb Incremental backup is working
d2301fc Incremental backup - in progress
c175059 Fixing NumberOfAllocatedPages in PureMemoryJournalWriter
ee55e1d Full backup is working
f9d0195 FoundationDB
dacf58d Make transaction explicit
bf5661e A bit safer?
201e0d6 Make sure that we update version even in optimized same size scenario
8006008 Restore the DirectAdd marker when moving between single value mutli tree and multiple values multi tree
19bafd8 Fixing memory overwrite issue
4743b4b Moving to a non copying pure memory pager, to make sure that the pointer references that we use are always valid for the lifetime of the pager.
27802af crlf
87ab5fc Add debug method
f59d218 Better method name
67ba3f5 FoundationDB
269f558 Merge branch 'voron' of github.com:ayende/raven.voron into voron
cdb7229 During investigation
ff1df65 Fixing tests
47b7aa3 Revert "Write a tree root header when new tree is created"
9bb7c30 Better approach
e815513 Write a tree root header when new tree is created
cbaf491 * Making sure that we read using a separate handle (so we can take advantage of buffering. * Can handle journal EOF
f81f45c Make sure that we are actually flushing the data pager to disk on flushing
968e2f0 Making sure that the data pager is a memory mapped one
66e64cd Forgot to actually add pagesToWrite
5c14612 Adding early warning system
f02f213 Removing debug code
8b98f75 Avoid concurrency errors in setting this
e2bb1ca Make sure that we actually read both headers from disk
3800465 Even more checks
96b1a46 Fixing wrong position send to journal writer
c48fb8d Fixing various issues with managing memory during recovery
586ec46 Making sure we have the right permissions & sharing options for opening a pager while the journal is ready for writes
95b4dde Make sure that we call add ref on the scratch pager.
32b6ef2 Increase size of work that we are allowed to merge.
7dbed25 Better behavior when we do the write in a sync manner
00eaedd Various fixes to complete the previous work
3fbe244 DeleteOnClose for journal files
6a9f210 * Flushing to log should not actually touch the journals, we are already dealing with them in the scratch, and we should have the page table in memory * Free scratch pages * Will read (just tx header) from the log file.
34a37e8 Will now track both the position inside the journal file and the position in the scratch file
eed9a17 LOTS of changes, and probably non working code. The basic idea is that I am going to do the following:
e3de010 Changing the way we are handling headers files. We'll use direct I/O to handle them, instead of relying on mmap files. Will also force a flush on write. Exposed both large & fine grained API for handling this.
722ad52 Merge branch 'voron' of github.com:ayende/raven.voron into voron
204c6dc Proper read/write locks on getting snapshot from journal file
0c1920e Make sure that we actually clear the page table, not just pretend to!
ea4fb30 File headers are stored in a separate small files instead first two pages of the data file
d5b22fc added parallel writes and reads to lmdb_perf_test
7e2ebf3 minor fixes in leveldb_perf_tests
58dad0c Merge branch 'voron' of https://github.com/ayende/raven.voron into voron
0477f9b Slight more efficient way to filter transactions Debug code that will need to be removed
49430cf Better debut experience
95c46ad Run the test using file i/o and not memory
c055800 Better perf tests
8382650 added parallel writes/reads for leveldb_perf_test
38dc169 Will no longer require the usage of performance counters
09007e8 Minor
53bceee Adding c++11 flags to compiler

git-subtree-dir: Raven.Voron
git-subtree-split: 538de072381c1e2560a2bd35217226d853bbed75
Merge commit 'e78143634bca8db2e65e6a2af9f77c45fe8ab3c9' into New3

Conflicts:
	Raven.Voron/Performance.Comparison/Other/LMDB/Makefile
	Raven.Voron/Performance.Comparison/Other/LMDB/lmdb_perf_test.cpp
	Raven.Voron/Performance.Comparison/Other/leveldb_perf_test.cc
	Raven.Voron/Performance.Comparison/Performance.Comparison/Constants.cs
	Raven.Voron/Performance.Comparison/Performance.Comparison/Esent/EsentTest.cs
	Raven.Voron/Performance.Comparison/Performance.Comparison/LMDB/LmdbTest.cs
	Raven.Voron/Performance.Comparison/Performance.Comparison/PerfTracker.cs
	Raven.Voron/Performance.Comparison/Performance.Comparison/Performance.Comparison.csproj
	Raven.Voron/Performance.Comparison/Performance.Comparison/Program.cs
	Raven.Voron/Performance.Comparison/Performance.Comparison/SQLCE/SqlCeTest.cs
	Raven.Voron/Performance.Comparison/Performance.Comparison/SQLServer/SqlServerTest.cs
	Raven.Voron/Performance.Comparison/Performance.Comparison/SQLite/SqlLiteTest.cs
	Raven.Voron/Performance.Comparison/Performance.Comparison/Voron/VoronTest.cs
	Raven.Voron/Voron.Benchmark/Voron.Benchmark.csproj
	Raven.Voron/Voron.Tests/Backups/Full.cs
	Raven.Voron/Voron.Tests/Backups/Incremental.cs
	Raven.Voron/Voron.Tests/Bugs/MultiAdds.cs
	Raven.Voron/Voron.Tests/Bugs/PageSplitter.cs
	Raven.Voron/Voron.Tests/Bugs/Recovery.cs
	Raven.Voron/Voron.Tests/Bugs/RecoveryMultipleJournals.cs
	Raven.Voron/Voron.Tests/Journal/BasicActions.cs
	Raven.Voron/Voron.Tests/Journal/EdgeCases.cs
	Raven.Voron/Voron.Tests/Journal/Mvcc.cs
	Raven.Voron/Voron.Tests/Journal/UncommittedTransactions.cs
	Raven.Voron/Voron.Tests/Storage/BigValue.cs
	Raven.Voron/Voron.Tests/Storage/Pagers.cs
	Raven.Voron/Voron.Tests/Storage/Restarts.cs
	Raven.Voron/Voron.Tests/Storage/SplittingVeryBig.cs
	Raven.Voron/Voron.Tests/StorageTest.cs
	Raven.Voron/Voron.Tests/Trees/Basic.cs
	Raven.Voron/Voron.Tests/Voron.Tests.csproj
	Raven.Voron/Voron.Tryout/Program.cs
	Raven.Voron/Voron.Tryout/Voron.Tryout.csproj
	Raven.Voron/Voron/Debugging/DebugStuff.cs
	Raven.Voron/Voron/Debugging/EnvironmentStats.cs
	Raven.Voron/Voron/Debugging/TreeDumper.cs
	Raven.Voron/Voron/Impl/Backup/FullBackup.cs
	Raven.Voron/Voron/Impl/Backup/IncrementalBackup.cs
	Raven.Voron/Voron/Impl/FileHeaders/FileHeader.cs
	Raven.Voron/Voron/Impl/Journal/JournalFile.cs
	Raven.Voron/Voron/Impl/Journal/JournalInfo.cs
	Raven.Voron/Voron/Impl/Journal/JournalReader.cs
	Raven.Voron/Voron/Impl/Journal/LogSnapshot.cs
	Raven.Voron/Voron/Impl/Journal/WriteAheadJournal.cs
	Raven.Voron/Voron/Impl/PagerState.cs
	Raven.Voron/Voron/Impl/Paging/AbstractPager.cs
	Raven.Voron/Voron/Impl/Paging/FilePager.cs
	Raven.Voron/Voron/Impl/Paging/IVirtualPager.cs
	Raven.Voron/Voron/Impl/Paging/MemoryMappedPager.cs
	Raven.Voron/Voron/Impl/Paging/PureMemoryPager.cs
	Raven.Voron/Voron/Impl/StorageEnvironmentState.cs
	Raven.Voron/Voron/Impl/Transaction.cs
	Raven.Voron/Voron/Impl/TransactionMergingWriter.cs
	Raven.Voron/Voron/Impl/WriteBatch.cs
	Raven.Voron/Voron/StorageEnvironment.cs
	Raven.Voron/Voron/StorageEnvironmentOptions.cs
	Raven.Voron/Voron/Trees/Page.cs
	Raven.Voron/Voron/Trees/PageSplitter.cs
	Raven.Voron/Voron/Trees/Tree.cs
	Raven.Voron/Voron/Trees/TreeIterator.cs
	Raven.Voron/Voron/Util/AsyncManualResetEvent.cs
	Raven.Voron/Voron/Util/Crc.cs
	Raven.Voron/Voron/Util/DataCopier.cs
	Raven.Voron/Voron/Util/NativeFileMethods.cs
	Raven.Voron/Voron/Voron.csproj
	Raven.Voron/Voron/packages.config
update voron
postpone timebomb
fix test with the new sync async wrapper error handling