realXtend/naali — pull request #508 by jonnenauha (2012-07-31 18:05:04+00:00)
Title: Adminotech - AssetAPI asset bundle support + ArhivePlugin for zip bundle implementation

This pull request encapsulates two things. First proper support for asset bundles to AssetAPI and secondly ArchivePlugin that implements zip bundle support agains the new functionality in AssetAPI.

## AssetAPI

AssetAPI now has a consept of asset bundles. It understands what they are and that they are there to provide IAssets. `IAssetBundle` is a new interface that needs to be implemented when someone wants to introduce a new asset bundle type. There is also similarly `IAssetBundleTypeFactory` just like for IAssets have. These factories can be registered from 3rd party plugins (like ArchivePlugin). The factories create asset bundles that again provide sub assets.

AssetAPI has been modified to understand the difference of bundles and sub assets inside of them. This is totally transparent to the code that actually calls `AssetAPI::RequestAsset(...)`. Asset references in Tundra can now be bundle#subAsset eg. assets.zip#materials/my.material. When a request like this comes to the asset system it will detect that it first needs the asset bundle in front of the # separator. It will request the bundle normally via an asset transfer that will be materialized into the `IAssetBundle` with the bundle factory. Once the bundle has been loaded successfully all the pending sub asset requests to it will be loaded from the bundle and their asset transfers will succeed or fail depending on weather the bundle can provide them. Again this is totally transparent to 3rd party code that calls RequestAsset(bundle#subAsset) and the flow is exactly the same.

Relative references will work inside a sub asset like they do normally. If "assets.zip#materials/my.material" depends on `my.png` the context will be correctly resolved to "assets.zip#materials/" if the dependent sub asset is in the root it will resolve correctly to "assets.zip#".

**Feature summary**

* New `IAssetBundle` and `IAssetBundleTypeFactory` for plugins to implement and register their bundles.
* Fixed the hardcoding away from `AssetAPI::GetResourceTypeFromAssetRef`. This made AssetAPI aware of all the potentially supported asset types even if their factories were in 3rd party modules. This function is no longer static and it will query all the registered IAsset and IAssetBundle factories if they can handle the file extension of the requested reference. There was a long standing todo in the function that the type should be asked from the factories and not be hardcoded.
* Files local asset provider not to block "forever" if there are lots of failed requests. This made Tundra unresponsive if there was eg. 500 local refs to disk that did not exist for a long time. Essentially adds the same throtling check when the requests fail like it is in the succeeded code path.
* Assets windows (shift+a) has been modified to show all IAssetBundles just like we do with storages. It will show the zip and all sub assets loaded into memory from it under it. You can reload/unload/refetch/modify/open cach file etc. the sub assets with the right click menu as you would expect.

## ArchivePlugin

This new Tundra plugin provides asset bundle support for ZIP files. It is a optional module but my proposal is to make it part of the core distribution. In this pull request it is turned on by default for windows and can be enabled to linux/mac as soon as their build scripts satisfy the zziplib dependency. Windows full build script now has support for building zziplib and zlib (dependency for zziplib) from sources.

ZIP file support is important for Tundra as we start to get bigger and bigger scenes. Lets take a example where we have 600 http asset references to the web. Now we already have our AssetAPI and AssetCache aware of last modified checks and it will only download the files from the web if they have been changed. But the last modified is always done to the server never the less (so we know if it has changed). Jukka and friends at Ludo have already shows some measurements that show that AssetAPI takes significant time to process these in our example 600 last modified web requests, its a huge amount of web requests even if they are very small in size. ZIP support mitigate this problem to as many last modified requests as there are zip bundles in the scene. If we pack all the static content that is not expected to change when the scene is ready to a single zip file, we end up with one request check, no re-download for the zip from the web, threaded unpacking of the zip file and asynch loading (for those types that support it eg. ogre assets) for the sub assets.

* Registers "Archive" type bundle factory to AssetAPI.
* Deserializes the zip to asset cache in a worker thread that wont block the main loop.
* Provides sub asset data from the bundle to AssetAPI to make the IAssets.