Opening development branch for logical addressing in FSFS

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1509279 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch:  Provide an updated BRANCH-README.
We inherited that file from our parent branch 'fsfs-improvements'.

* BRANCH-README
  (): Describe purpose and goals for this branch

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1509296 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch:  Bump FSFS format number and introduce
the new "addressing" option to fsfs format files.  Make that available
to our internal code through the new svn_fs_fs__use_log_addressing API.
IOW, we support mixed addressing repos from the beginning.

* subversion/libsvn_fs_fs/fs.h
  (SVN_FS_FS__FORMAT_NUMBER): bump to 7
  (SVN_FS_FS__MIN_LOG_ADDRESSING_FORMAT): declare new feature level
  (fs_fs_data_t): add start rev for logical addressing

* subversion/libsvn_fs_fs/fs_fs.c
  (read_format,
   svn_fs_fs__write_format): read / write the new addressing option
  (svn_fs_fs__open): update caller
  (upgrade_body): ditto; enable log addressing for the next shard, if any
  (svn_fs_fs__create): handle the generic compatibility option that 
                       will be introduced with FSX;
                       init addressing option depending on it
  (svn_fs_fs__info_format): update to accept f7

* subversion/libsvn_fs_fs/util.h
  (svn_fs_fs__use_log_addressing): declare new private API

* subversion/libsvn_fs_fs/util.c
  (svn_fs_fs__use_log_addressing): implement

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1509342 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch: Follow-up to r1509342.

* subversion/tests/libsvn_fs_fs/fs-pack-test.c
  (write_format): Write complete f7 format files.

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1509605 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch: Follow-up to r1509605.

* subversion/tests/libsvn_fs_fs/fs-pack-test.c
  (write_format): don't write duplicate lines to format files.

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1509608 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch:  Return an error when the format file
specifies logical addressing in a non-shared repo.  Just disabling
logical addressing in that case could mask more severe issues.

* subversion/libsvn_fs_fs/fs_fs.c
  (read_format): create an error if layout & addressing are inconsistent

Suggested by: danielsh

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1509609 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch:  Follow-up to r1509609.

* subversion/libsvn_fs_fs/fs_fs.c
  (read_format): fix typo commentary and error string;
                 fix missing indirection in error condition;
                 fix parameters provided to error message

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1509627 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch:  Fix / handle  more compatibility cases.

* subversion/libsvn_fs_fs/fs.h
  (SVN_FS_FS__FORMAT_NUMBER): direct devs to svn_fs_fs__create
   in case of format bumps

* subversion/libsvn_fs_fs/fs_fs.c
  (svn_fs_fs__create): reject 1.0 compatibility;
                       fix format version used for 1.8 compat

* subversion/libsvn_fs_fs/fs.c
  (initialize_fs_struct): initialize / default to "phys addressing"

Suggested by: danielsh

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1509695 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch:  Fix the hotcopy for f7 repos.

* subversion/libsvn_fs_fs/fs_fs.c
  (svn_fs_fs__open): actually use the min_log_addressing_rev value
                     read from the format file
* subversion/libsvn_fs_fs/hotcopy.c
  (hotcopy_create_empty_dest): copy min_log_addressing_rev as well

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1509732 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch:  Introduce the notion of "item_index" as a
generalized replacement for "offset" that works in both adressing modes.

This requires 3 sets of changes
- rename "offset" to "item_index" in structs and some local variables
- hange the FSFS ID API to say "item" instead of "offset"
- translate the item_index into actual file offsets before accessing
  the respective file

For the latter, we extend the existing svn_fs_fs__item_offset function
to take an optional TXN_ID parameter to make it suitable for general
offset translation.  The current implementation remains simple because
we don't really have logical addresses, yet.  Use that mapping API also
in those few places where we don't do so already.

* subversion/libsvn_fs_fs/id.h
  (svn_fs_fs__id_part_t): update commentary
  (svn_fs_fs__id_rev_offset,
   svn_fs_fs__id_offset): rename these to ...
  (svn_fs_fs__id_rev_item,
   svn_fs_fs__id_item): ... these, respectively
  (svn_fs_fs__id_rev_create): rename parameter

* subversion/libsvn_fs_fs/id.c
  (fs_fs__id_t): rename rev_offset member to rev_item
  (svn_fs_fs__id_rev_offset,
   svn_fs_fs__id_offset): rename these to ...
  (svn_fs_fs__id_rev_item,
   svn_fs_fs__id_item): ... these, respectively; update struct user
  (svn_fs_fs__id_rev,
   svn_fs_fs__id_unparse,
   svn_fs_fs__id_txn_create_root,
   svn_fs_fs__id_txn_create,
   svn_fs_fs__id_parse): update struct user
  (svn_fs_fs__id_rev_create): ditto; update signature

* subversion/libsvn_fs_fs/util.h
  (svn_fs_fs__item_offset): update signature; add TXN_ID parameter

* subversion/libsvn_fs_fs/util.c
  (svn_fs_fs__item_offset): update

* subversion/libsvn_fs_fs/fs.h
  (representation_cache_key_t,
   window_cache_key_t,
   fs_fs_data_t,
   representation_t): replace offsets with 64 bit item_index

* subversion/libsvn_fs_fs/low_level.h
  (svn_fs_fs__rep_header_t): rename base_offset to base_item_index

* subversion/libsvn_fs_fs/low_level.c
  (svn_fs_fs__parse_representation,
   svn_fs_fs__unparse_representation,
   svn_fs_fs__read_rep_header,
   svn_fs_fs__write_rep_header): update struct users

* subversion/libsvn_fs_fs/cached_data.c
  (dbg_log_access,
   open_and_seek_revision): update signature; struct usage and offset translation
  (open_and_seek_transaction): translate offset before accessing the file
  (open_and_seek_representation): update struct usage
  (get_node_revision_body,
   svn_fs_fs__get_node_revision): update API caller
  (rep_state_t): rename offset to item_index
  (auto_set_start_offset): update API caller
  (create_rep_state_body): update struct usage and offset translation
  (svn_fs_fs__rep_chain_length,
   get_window_key,
   build_rep_list,
   read_delta_window,
   svn_fs_fs__get_contents,
   svn_fs_fs__try_process_file_contents,
   svn_fs_fs__get_file_delta_stream,
   locate_dir_cache,
   svn_fs_fs__get_proplist): update struct usage

* subversion/libsvn_fs_fs/fs_fs.c
  (svn_fs_fs__noderev_same_rep_key): ditto

* subversion/libsvn_fs_fs/pack.c
  (compare_dir_entries_format6): update API caller

* subversion/libsvn_fs_fs/recovery.c
  (recover_find_max_ids,
   svn_fs_fs__find_max_ids): translate offsets before usage

* subversion/libsvn_fs_fs/rep-cache.c
  (svn_fs_fs__walk_rep_reference,
   svn_fs_fs__get_rep_reference, 
   svn_fs_fs__set_rep_reference): update struct users

* subversion/libsvn_fs_fs/transaction.c
  (rep_write_get_baton,
   rep_write_contents_close,
   write_hash_rep,
   write_hash_delta_rep,
   write_final_rev,
   commit_body): ditto

* tools/server-side/svn-rep-sharing-stats.c
  (record): ditto

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1509766 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch:  Add FSFS config file option to set
the block size in which we access rev / pack files.  For now, we
use it with aligned_seek only but it will soon also determine the
size of the blocks we parse.

* subversion/libsvn_fs_fs/fs.h
  (CONFIG_SECTION_IO,
   CONFIG_OPTION_BLOCK_SIZE): declare new config file elements
  (fs_fs_data_t): add block size

* subversion/libsvn_fs_fs/fs_fs.c
  (read_config): read the new option
  (write_config): extend config file template

* subversion/libsvn_fs_fs/cached_data.c
  (aligned_seek): take FS as parameter to select block size
  (open_and_seek_revision,
   open_and_seek_transaction,
   get_fs_id_at_offset,
   get_root_changes_offset,
   auto_read_diff_version,
   create_rep_state_body,
   read_delta_window,
   read_plain_window,
   get_contents,
   svn_fs_fs__get_changes): update callers


git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1509843 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch:  Document the new addressing scheme
of format7 in our structure doc.  Add the description of the index
file format as as separate document.  No functional change.

* subversion/libsvn_fs_fs/structure
  (Layout of the FS directory): add index files
  (Filesystem formats): add f7
  (Filesystem format options): document "addressing" option
  (Addressing modes,
   Index files): new sections
  (Packing revisions,
   Node-revision IDs,
   Revision file format): update
  (Transaction layout): update; add proto index files 

* subversion/libsvn_fs_fs/structure-indexes
  (): new file describing the binary index files

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1510990 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch: sync with the parent branch up to r1511149.
Resolve a minor conflict in fs-fs-pack-test.c.

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1511151 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch:  Factor out a function evaluating
the various version compatibility FS configuration parameters.
Use that function in all 3 backends such that all of them handle
all compat options that we have.

Allow for 1.x compatibility only and let the backend select the
FS format based on the minor version.

* subversion/include/private/svn_fs_util.h
  (svn_fs__compatible_version): declare new private API function

* subversion/libsvn_fs_util/fs-util.c
  (svn_fs__compatible_version): implement it

* subversion/libsvn_fs_fs/fs_fs.c
  (svn_fs_fs__create): use the new API

* subversion/libsvn_fs_base/fs.h
  (SVN_FS_BASE__FORMAT_NUMBER): remind people to update *_create upon bump

* subversion/libsvn_fs_base/fs.c
  (base_create): use the new API

* subversion/libsvn_fs_x/fs.h
  (SVN_FS_X__FORMAT_NUMBER): remind people to update *_create upon bump

* subversion/libsvn_fs_x/fs_x.c
  (svn_fs_x__create): use the new API

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1511324 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch:  Follow-up to r1511324.

Address various compatibility check issues.  First, the compat
version precedence check has to be reversed.  Second, we should
select the oldest version of these:

- current tool version (takes care of wrong major versions as well)
- SVN_FS_CONFIG_COMPATIBLE_VERSION
- SVN_FS_CONFIG_PRE_1_*_COMPATIBLE

Also, we want the patch level to have a defined value.  We set
it to 0 as there must be no format change in a patch release.

* subversion/include/private/svn_fs_util.h
  (svn_fs__compatible_version): update docstring

* subversion/libsvn_fs_util/fs-util.c
  (add_compatility): new utility to combine version requirements
  (svn_fs__compatible_version): check versions in proper order;
                                limit result to current version;
                                set patch level to 0

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1511346 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch:  Provide a test case for the new
svn_fs__compatible_version function.

* subversion/tests/libsvn_fs/fs-test.c
  (test_compat_version): new test
  (test_funcs): register the new test

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1511363 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch:  More fixes to svn_fs__compatible_version.
The version comparison was broken and version->tag was not well-defined.

* subversion/include/private/svn_fs_util.h
  (svn_fs__append_to_merged_froms): update docstring

* subversion/libsvn_fs_util/fs-util.c
  (add_compatility): fix version comparison
  (svn_fs__compatible_version): give tag member a defined value

Found by: danielsh

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1511364 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch:  Implement the packed_stream_t object
used to read 7b/8b encoded data as used in the format 7 index.
This is committed separate from its usage to make review easier.

* subversion/libsvn_fs_fs/index.c
  (): new file
  (MAX_NUMBER_PREFETCH): new constant
  (value_position_pair_t,
   packed_number_stream_t): packed stream data structures
  (stream_error_create,
   packed_stream_read,
   packed_stream_open,
   packed_stream_close,
   packed_stream_get,
   packed_stream_seek,
   packed_stream_offset): implement packed stream logic
  (encode_uint,
   encode_int,
   decode_int): data conversion utilities

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1511365 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch: Follow-up to r1511365.
Doc string only change.

* subversion/libsvn_fs_fs/index.c
  (): add some commentary

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1511368 13f79535-47bb-0310-9956-ffa450edef68
On the log-addressing branch: Follow-up to r1510990, documenting
another in-txn file used by format7.

* subversion/libsvn_fs_fs/structure
  (Transaction layout): mention the "itemidx" file

git-svn-id: https://svn.apache.org/repos/asf/subversion/branches/log-addressing@1511382 13f79535-47bb-0310-9956-ffa450edef68