RealworldSystems/swBuildScripts — Issue #7: add source tree relocation to documentation
Author: tiesmaster
Created: 2011-12-08 13:04:32+00:00
State: open

See https://gemini.realworld-systems.com/issue/ViewIssue.aspx?id=7563&PROJID=47 for more information about this. Here is the text from the issue:

With SAB, we've introduced a new way of making it possible to relocate a source tree, after the images have been built. This documentes how that works.

Background
In most programming languages, the build artifacts don't contain absolute paths, and hence, can easily be packaged and deployed, if wanted. For example, a C-program compiles to an EXE-file, and is usually agnostic about from where it is run, with Java JAR-files, this is similar. And so on.

With Smallworld, this is not the case. A Smallworld image contains absolute paths to the products that have been added, the method finder file contains the absolute paths to the directories of paths with source files, and finally, image files which NOT have been fused/joined have hard links to the swaf image (if that's the base image).

Best practices predicts to rebuild images anyhow if you deploy the software product to a new location, however, in certain situations (such as nightlies), we do want to copy our built artifacts to a location on the network for testing purposes. With our solution, this is possible, and it is divided in three parts: redirect sources, relocate products, and save deployable image.

Redirect Sources
The method finder holds the location for each source file, and during a build it is notified of new locations. When we build an image, the absolute paths are recorded by the method finder, which prevents us from relocating a path with sources in it.

The sources of Smallworld don't have this problem, since those are stored using the environment variable $SMALLWORLD_GIS, instead of absolute paths. The Class Browser in Emacs automatically resolves those according to the environment, making them portable.

It is unclear how they prevent absolute paths from entering the image, or if they change the paths afterwards, but this solution implements the latter option. The undocumented method finder interface has two methods (list_directories, and change_directory) and we utilize that to replace the absolute paths with relatives ones based on the environment variable $PROJECT_DIR. The build scripts always set that environment variable, so in Emacs, that variable will always have been set, making it possible for the Class Browser to resolve it.

The main logic lives in $swBuildScripts/bin/share/normalise_sources.magik.

Relocate Products
Smallworld stores the absolute path when products are added to the session, and when the image is stored, the absolute paths are in the image. This cannot be prevented, Smallworld's products are also stored with absolute paths. When Smallworld starts, it calls the method fixup_products(), which scans the SW_PRODUCTS_PATH array, and updates all existing products according to the new locations in the array. The SW_PRODUCTS_PATH always needs to contain the $SMALLWORLD_GIS path, and at startup the paths of the core modules are automatically fixed when the fixup_products()is called.

A downside to using the SW_PRODUCTS_PATH is that Smallworld will always add all products in the products path at startup, which is undesirable. The build system should be able to decide when the products are loaded. With UTRM for example, we first add only the runtime products (such as cambridge, munit, CDH), and in the second image, we add the project modules. Next, the CDH also requires that the products haven't been added by Smallworld at startup.

So we cannot just add our products to the SW_PRODUCTS_PATH, instead we correct the path of the top level product ourselves using the same logic of fixup_products(), letting that method do the rest. Smallworld calls this method very early at image startup, triggering a lot of warnings about missing product locations. Since this is confusing, we fix this very early at image restore time, by utilizing the image_restore_acps event of object sw:system.

For now, this solution requires you to have a top level product in the root of your source project, that has been added to Smallworld. However, an alternative solution is to store the PROJECT_DIR value at build time, and correct all project paths at image restore time, if it's different.

The main logic lives in $swBuildScripts/bin/share/source_tree_relocation.magik

Absolute Backlinks
Smallworld tries to store relative paths for its backlinks, if possible. However, that is not always possible and then absolute paths are set for the backlinks. SAB-7846 implementes a solution for that, see that issue for details.

How to Enable this
To install Source Tree Relocation, set the pre image save procedure to install_source_tree_relocation(), eg.:
magik_image.register_new("test_image",
:image_file_name, "test_image",
:pre_save_image_proc, install_source_tree_relocation
)

The feature is enabled by setting the variable ROOS_ENABLE_SOURCE_TREE_RELOCATION

eg using the environment.bat:
set ROOS_ENABLE_SOURCE_TREE_RELOCATION=true

eg using the swBuildScripts Rakefile:
rake default ROOS_ENABLE_SOURCE_TREE_RELOCATION=true