pioneerspacesim/pioneer — pull request #989 by johnbartholomew (2012-02-18 21:46:12+00:00)
Title: File system abstraction

[for review/testing -- this patch touches a lot of code and has certainly introduced bugs; it also affects the build/run process under VS2010 and VS2008 hasn't been updated yet]

This patch series adds a new API for reading data files, enumerating directory contents, finding the application's data and configuration directories and creating new directories.

The intentions are:

* Search for data files in multiple locations (the application's installed data directory and a per-user data directory). This is primarily for #124.
* In the future, allow data files to be loaded from bundle/archive formats like .zip.
* In the future, allow data files to be loaded by memory mapping (where appropriate).
* In the future, allow a more comprehensive virtual resource hierarchy.
* Allow a direct win32 api implementation for operations where POSIX or C runtime functions aren't quite good enough (e.g., to support paths containing arbitrary unicode characters)

What's missing:

* The path handling functions are rather inadequate at the moment; they need to be made more robust.
* Proper path safety: e.g., at the moment you can call ReadFile with an absolute path or a relative path containing ".." and thereby escape from the specified data search directories.
* Sensible error handling (this is missing from almost the entire Pioneer code-base though)
* No streaming functionality for files -- FileSystem always reads files in their entirety. This is usually desirable anyway so I'm not too fussed, but streaming may be useful at some point. (note: if you want to implement this, please read [Buffer Centric IO](http://fgiesen.wordpress.com/2011/11/21/buffer-centric-io/) first for a description of a fairly good streaming API; it's not perfect but it's worth considering)
* No file writing functionality (not even the API for it). Various pieces of code want to write files (model cache, save files, log files, screenshots); at the moment this just goes through the same code it always has. Save files in particular would benefit from going through the FileSystem API since people may insert weird unicode characters in their save file names.
* A few minor things still don't load their data through FileSystem (e.g. Pioneer's icon image, pidebug.lua, and a few others that I've forgotten and will have to find again).
* Various bits of code that now load through FileSystem could do with further clean-up.

Extra design notes / open issues:

* Because FileSystem loads a file in a block and then gives you a blob of data, various bits of file reading code have had to be rewritten to work from a memory blob rather than a `FILE*` stream. I've added a couple of helper objects for this: `StringRange`, which keeps track of a slice within a text blob and provides some (very simple) text scanning functionality, and `ByteRange` which keeps track of a slice within a data blob and provides a read() function that looks like fread(). I think the general idea/purpose of these objects is right, but the details could do with cleaning up (e.g., `ByteRange` could get data-type specific reading functions and be used underneath the serialization code, `StringRange` could learn how to process utf-8 text or provide nicer scanning functions; at a minimum I'd like to improve the naming of `StringRange`'s methods).
* On win32, pioneer now looks for its data files underneath the executable's directory rather than the current working directory. This means that the executable has to be in the right location for the game to find its files. To this end I've modified the VS2010 project to output the executable to $(SolutionDir)../../, and changed the output names for the Debug and PreRelease configurations so they don't clobber each other. However, the .dlls in win32/libs need to be moved manually so they can be found. I'd like to get to a stage where you can build and run from VS with no changes to the projects (including no changes to the per-user project config) and it'll work, but I'm not sure exactly where we want everything to be. Suggestions from the people who develop primarily in Windows would be welcome.