Development Environment for Multiple Viewers

== Introduction ==
Highlights:
* Productivity increase (automation of repetitive tasks)
* Complete directory structure proposal
* Directory Dependent History and Environment Switching
* Support for development of multiple viewers on one host
* quilt
* ctags
== Directory structure ==
=== The ROOT ===
Choose a directory that will contains everything related to SecondLife, OpenSim etc.
In my case this directory contains (currently) 12 GB of data, so pick a place with space ;).
For the sake of copying and pasting commands, lets use an environment variable for this
'root' directory.
 export ROOT=/usr/src/secondlife
I use /usr/src/secondlife, but you could of course use something else, like
/opt/secondlife, or $HOME/SLroot or whatever.
=== $ROOT/viewers ===
The root directory will contain a subdirectory viewers.
 mkdir $ROOT/viewers
viewers will contain any data related to viewer development (viewer build system
configuration, patches, viewer source trees, viewer subprocesses (ie SLVoice), User Application Directories (AppDir),
documentation...) while $ROOT itself will contain everything else (things that are less close to developing the viewers,
like special/patched/custom libraries, custom scripts, uploaded assets (textures, sounds, animations), snapshots, third party projects related to SecondLife etc etc).
Purely as a suggestion, you could do
 mkdir $ROOT/blender $ROOT/gimp $ROOT/libomv $ROOT/textures $ROOT/wav $ROOT/Snapshots
=== Viewer project base ===
Each viewer (version) needs it's own directory, not just for the source tree, but also for meta data (patches, quilt cache, build system configuration, etc). The below deals with a single viewer version, but you can repeat it for multiple viewers.
For the sake of easy copy and pasting, lets give our first viewer version an environment variable too. For example,
 export VIEWERNAME=snowglobe-1.x-svn
or
 export VIEWERNAME=snowglobe-2.x-svn
Then, for each such viewer create the following directories:
 cd $ROOT/viewers
 # And for each of the viewers:
 mkdir $VIEWERNAME
 cd $VIEWERNAME
 mkdir include
 mkdir timestamps
 mkdir quilt
 mkdir quilt/patches
 mkdir quilt/pc
The real source tree (the one containing the indra directory) should be called 'linden' and also goes here.
For example (in snowglobe-1.x-svn),
 svn checkout https://svn.secondlife.com/svn/linden/projects/2009/snowglobe/trunk linden
or (in snowglobe-2.x-svn)
 svn checkout https://svn.secondlife.com/svn/linden/projects/2010/snowglobe/trunk linden
Note: See Version_control_repository for a complete overview of available repository url's.
== Directory dependent history and environment ==
Now to automate everything! First install cdeh.
Also set DEFAULTPATH in your .bashrc. For example,
 export DEFAULTPATH="$HOME/bin:/usr/local/bin:/bin:/usr/bin:/sbin:/usr/sbin" # Or whatever your current PATH is!
 export PATH="$DEFAULTPATH"
Obviously, DEFAULTPATH may not depend on PATH.
If you intended to use cdeh also for library projects (ie, libraries in $ROOT) and you have multiple
compilers installed (ie, you use a special compiler version for the viewer) then you also should set
DEFAULT_TOPPROJECT in your .bashrc. Set it to $ROOT/viewers, but expand $ROOT (type it out fully) because obviously ROOT is not set at the moment you log in.
Here it might be wise to reboot, so that every login shell has DEFAULTPATH set; or otherwise make sure
to source .bashrc to set it until you have time to reboot. The effect of not having it set is getting
an empty PATH so that you will have to type full paths to execute anything :p
Next install this file in the root (/) of your operating system,
world readable, and rename it to /env.source. Yes, that is the real root, not $ROOT.
 cd $ROOT
 wget http://www.xs4all.nl/~carlo17/howto/root-env.source
 sudo cp root-env.source /env.source
 sudo chmod a+r /env.source
 rm root-env.source
Congratulations! You now have directory dependend history and environment switching!
Lets start with creating a separate history file for most of the core directories:
 cd $ROOT
 addhistory
 cd viewers
 addhistory
 # And for each of the viewers:
 cd $VIEWERNAME
 addhistory
=== Creating the environment files ===
Next, we have to add a few environments.
For each file, copy the entire block of text and paste it in your shell.
(This is needed for two reasons: this wiki doesn't allow to attach files,
and this way we can expand your already set environment variables.)
==== viewers/env.compiler ====
<nowiki>
cd $ROOT/viewers
cat << EOF > env.compiler
CC="gcc"
CXX="ccache g++"
export CC CXX
EOF
</nowiki>
==== viewers/env.viewers ====
<nowiki>
cd $ROOT/viewers
cat << EOF >> env.viewers
source /env.source
source $TOPPROJECT/env.compiler
# FIXME: correct those paths!
INSTALL_PREFIX="/sl" # Install prefix used for thirdparty libraries.
CCACHE_DIR=/ramdisk/ccache # Where you want to keep your ccache cache.
# This is needed for 2.1 and higher. You need to get breakpad and compile the
# dump_syms tool for your operating system. Then have this point to the executable.
DUMP_SYMS=/usr/src/secondlife/breakpad/trunk/src/tools/linux/dump_syms/dump_syms
LDFLAGS+=" -L$INSTALL_PREFIX/lib -L$INSTALL_PREFIX/usr/lib"
PKG_CONFIG_PATH="$INSTALL_PREFIX/lib/pkgconfig"
pre_path "$INSTALL_PREFIX/bin"
pre_path "$INSTALL_PREFIX/lib" LD_LIBRARY_PATH
pre_path "$INSTALL_PREFIX" CMAKE_PREFIX_PATH
pre_path "$INSTALL_PREFIX/usr" CMAKE_PREFIX_PATH
pre_path "$PROJECTBASE/include" CMAKE_INCLUDE_PATH
export PKG_CONFIG_PATH LD_LIBRARY_PATH CMAKE_PREFIX_PATH CCACHE_DIR CMAKE_INCLUDE_PATH
REPOBASE="$PROJECTBASE/linden"
# Need export, because this is used by SLVoice
export INDRA="$REPOBASE/indra"
# Automatically read the generated ctags file, and set the tabstop to 4.
alias vi='vim -c "set tags=$INDRA/$BUILDDIR/tags" -c "set tabstop=4"'
# This can be used to grep all sources. For example: grep LLSelectMgr `s`
alias s='/bin/ls $INDRA/ll*/*.{cpp,h} $INDRA/linux_crash_logger/*.{cpp,h} $INDRA/lscript/*/*.{cpp,h} $INDRA/newview/*.{cpp,h} $INDRA/media_plugins/*/*.{cpp,h}'
# This can be used to grep all cmake/configure related files. For example: grep WEBKIT_PLUGIN_LIBRARIES `c`
alias c='/usr/bin/find $INDRA -path "$INDRA/viewer-*" -prune -o \( -name CMakeLists.txt -o -name '"'"'*.cmake'"'"' \) -print'
alias configure='source $PROJECTBASE/env.source; make configure'
alias cdr='cd $PROJECTBASE'
alias cdl='cd $PROJECTBASE/linden'
alias cdi='cd $INDRA'
alias cdb='cd $INDRA/$BUILDDIR'
alias cdn='cd $INDRA/$BUILDDIR/newview'
alias cdp='cd $INDRA/$BUILDDIR/newview/packaged'
QUILT_PATCHES="$PROJECTBASE/quilt/patches"
QUILT_PATCHES_PREFIX=1
QUILT_DIFF_OPTS="-p"
export QUILT_PATCHES QUILT_PATCHES_PREFIX QUILT_DIFF_OPTS
function make ()
{
 CURDIR=$(pwd)
 cd $INDRA
 case $@ in
 configure)
 (cd $PROJECTBASE/include && ln -sf ../linden/libraries/include/tut)
 rm -f $BUILDDIR/CMakeCache.txt
 ./develop.py $CONFIGURE_OPTS configure $CMAKE_DEFS -DCMAKE_CXX_FLAGS:STRING="$CXXFLAGS" -DCMAKE_EXE_LINKER_FLAGS:STRING="$LDFLAGS"
 cd $BUILDDIR/newview && (\
 ln -sf ../llplugin/slplugin/SLPlugin; \
 mkdir -p llplugin; \
 ln -sf ../../media_plugins/webkit/libmedia_plugin_webkit.so llplugin/libmedia_plugin_webkit.so; \
 ln -sf ../../media_plugins/gstreamer010/libmedia_plugin_gstreamer010.so llplugin/libmedia_plugin_gstreamer.so; \
 ln -sf $INDRA/newview/app_settings; \
 ln -sf $INDRA/newview/fonts; \
 ln -sf $INDRA/newview/skins; \
 ln -sf $INDRA/newview/character; \
 ln -sf $INDRA/newview/res-sdl; \
 ln -sf $INDRA/newview/gpu_table.txt; \
 ln -sf $INDRA/newview/featuretable_linux.txt; \
 ln -sf $INDRA/newview/linux_tools/register_secondlifeprotocol.sh; \
 ln -sf $INDRA/newview/linux_tools/launch_url.sh; \
 ln -sf $INDRA/newview/linux_tools/handle_secondlifeprotocol.sh; \
 ln -sf $INDRA/newview/secondlife-i686.supp; \
 ln -sf $INDRA/newview/res/snowglobe_icon.png; \
 ln -sf $REPOBASE/../../gdbinit .gdbinit; \
 ln -sf $TOPPROJECT/SLVoice; \
 cd app_settings; \
 ln -sf $REPOBASE/scripts/messages/message_template.msg; \
 ln -sf $REPOBASE/etc/message.xml; \
 test -d ../skins/default/xui/en-us && \
 cd ../skins/default/xui/en-us && ln -sf mime_types_linux.xml mime_types.xml; \
 )
 mkdir -p $REPOBASE/libraries/$(uname -m)-linux/bin && cd $REPOBASE/libraries/$(uname -m)-linux/bin && ln -sf $DUMP_SYMS
 ;;
 "")
 ./develop.py $CONFIGURE_OPTS build
 ;;
 clean|realclean)
 if test -L $BUILDDIR; then
 rm -rf $BUILDDIR/*;
 else
 ./develop.py $CONFIGURE_OPTS clean
 fi
 ;;
 ctags|tags)
 ctags -f $BUILDDIR/tags `s`
 ;;
 *)
 echo "Don't know how to make $@"
 ;;
 esac
 cd $CURDIR
}
function store_timestamps ()
{
 CURDIR=$(pwd)
 cd $REPOBASE
 for f in `find . -name '*.h' -o -name '*.cpp' -o -name '*.cc' -o -name CMakeLists.txt -o -name '*.cmake'`; do
 mkdir -p `dirname $PROJECTBASE/timestamps/$f`;
 md5sum $f > $PROJECTBASE/timestamps/$f;
 touch -r $f $PROJECTBASE/timestamps/$f;
 done
 cd $CURDIR
}
function restore_timestamps ()
{
 CURDIR=$(pwd)
 cd $REPOBASE
 for f in `find . -name '*.h' -o -name '*.cpp' -o -name '*.cc' -o -name CMakeLists.txt -o -name '*.cmake'`; do
 if test "`cat $PROJECTBASE/timestamps/$f`" = "`md5sum $f`"; then
 touch -r $PROJECTBASE/timestamps/$f $f;
 fi
 done
 cd $CURDIR
}
function quilt ()
{
 CURDIR=$(pwd)
 cd $REPOBASE
 case $@ in
 refresh)
 /usr/bin/quilt refresh -p0
 ;;
 *)
 /usr/bin/quilt "$@"
 ;;
 esac
 cd $CURDIR
}
function svn ()
{
 CURDIR=$(pwd)
 cd $REPOBASE
 case $@ in
 update)
 if /usr/bin/quilt top; then
 echo "Storing timestamps..."
 store_timestamps;
 echo "Merging update..."
 /usr/bin/quilt diff -p0 -z > current.$$.diff && \
 patch -p0 -R < current.$$.diff && \
 TOP=`/usr/bin/quilt top` && \
 /usr/bin/quilt pop -aR && \
 /usr/bin/svn update && \
 /usr/bin/quilt push "$TOP" && \
 patch -p0 < current.$$.diff && \
 rm current.$$.diff;
 echo "Restoring timestamps..."
 restore_timestamps;
 else
 /usr/bin/svn update
 fi
 ;;
 diff)
 if /usr/bin/quilt top 2>/dev/null; then
 /usr/bin/quilt diff -p0 -z;
 echo "Patch relative from 'tag' \"$(basename `/usr/bin/quilt top`)\"."
 else
 /usr/bin/svn diff;
 fi
 ;;
 commit)
 if /usr/bin/quilt top 2>/dev/null; then
 echo "There are still quilt patches applied!"
 else
 /usr/bin/svn "$@"
 fi
 ;;
 *)
 cd $CURDIR
 /usr/bin/svn "$@"
 ;;
 esac
 cd $CURDIR
}
echo "CONFIGURE_OPTS = \"$CONFIGURE_OPTS\""
echo "CMAKE_DEFS = \"$CMAKE_DEFS\""
test -z "$CMAKE_PREFIX_PATH" || echo "CMAKE_PREFIX_PATH = \"$CMAKE_PREFIX_PATH\""
EOF
</nowiki>
==== viewers/gdbinit ====
<nowiki>
cd $ROOT/viewers
cat << EOF > gdbinit
file ./secondlife-bin
set print static-members off
# Examples:
# Make gdb find the right libraries.
#set env LD_LIBRARY_PATH /usr/src/secondlife/openjpeg-tools/openjpeg-1.3+dfsg:/sl/lib:/sl/usr/lib:/usr/local/lib:/usr/src/secondlife/viewers/snowglobe/snowglobe-1.x-svn/linden/indra/viewer-linux-x86_64-debug/llcommon
# Make it login to a different grid
#set args -loginuri="http://osgrid.org:8002" -loginpage "http://osgrid.org/loginscreen.php" -helperuri "http://osgrid.org/"
# Make gdb find the sources of libraries under test.
#dir /usr/src/secondlife/openjpeg-tools/openjpeg-1.3+dfsg:/usr/src/libc6/eglibc-2.9/nptl:/usr/src/secondlife/viewers/pthread_debug:/usr/src/debian/apr-1.3.5:/usr/src/debian/openal-soft-1.8.466
# Load gstreamer plugins from our prefix
#set env GTS_PLUGIN_PATH /sl/lib
# Do some hackerish threading debugging with preloaded libraries.
#set env LD_PRELOAD /usr/src/secondlife/viewers/pthread_debug/libpthread_debug.so /usr/lib/libapr-1.so.0 /sl/lib/libcwd_r.so/lib/libdl.so.2
# Run gdb on the monitor of a remote PC (ssh-ed into hikaru, where the viewer is installed and running).
#set env DISPLAY hikaru:0.0
# Etc etc. Set all environment variables needed for testing.
#set env __GL_YIELD NOTHING
#set env LIBCWD_NO_STARTUP_MSGS 1
EOF
</nowiki>
==== viewers/$VIEWERNAME/env.source ====
And last but not least for each of the viewers:
<nowiki>
cd $ROOT/viewers/$VIEWERNAME
echo "export TOPPROJECT=$ROOT/viewers" > env.source
echo "PROJECTBASE=\"\$TOPPROJECT/$VIEWERNAME\"" >> env.source
echo 'CONFIGURE_OPTS="--type=Debug -m64 --standalone" # FIXME' >> env.source
echo "BUILDDIR=\"viewer-linux-$(uname -m)-debug\" # FIXME" >> env.source
cat << EOF >> env.source
CMAKE_DEFS="-DLL_TESTS:BOOL=ON -DPACKAGE:BOOL=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON"
source "\$TOPPROJECT/env.viewers"
# FIXME: The user dir (with your SL cache and log files etc)
#export SECONDLIFE_USER_DIR=/usr/src/secondlife/viewers/OSUserAppDir.snowglobe2
# FIXME: Viewer version dependend libqtwebkit install.
#pre_path /usr/src/secondlife/llqtwebkit/install2/include CMAKE_INCLUDE_PATH
#pre_path /usr/src/secondlife/llqtwebkit/install2/lib CMAKE_LIBRARY_PATH
export CMAKE_INCLUDE_PATH CMAKE_LIBRARY_PATH
test -z "\$CMAKE_INCLUDE_PATH" || echo "CMAKE_INCLUDE_PATH = \"\$CMAKE_INCLUDE_PATH\""
test -z "\$CMAKE_LIBRARY_PATH" || echo "CMAKE_LIBRARY_PATH = \"\$CMAKE_LIBRARY_PATH\""
test -z "\$CXXFLAGS" || echo "CXXFLAGS = \"\$CXXFLAGS\""
test -z "\$LDFLAGS" || echo "LDFLAGS = \"\$LDFLAGS\""
EOF
</nowiki>
----
Now go edit those generated files and fix them for your personal taste.
Pay extra attention to the lines saying 'FIXME'!
== Using all of the above ==
So what's the result of all of this? Well, as soon as you change directory into $VIEWERNAME (or a subdirectory)
therefore, you should see cdeh print something like this:
 % cd snowglobe-1.x-svn/linden
 History changed from "/usr/src/secondlife/viewers" to "/usr/src/secondlife/viewers/snowglobe/snowglobe-1.x-svn"
 Environment changed from "/" to "/usr/src/secondlife/viewers/snowglobe/snowglobe-1.x-svn"
 CONFIGURE_OPTS = "--type=Debug -m64 --standalone"
 CMAKE_DEFS = "-DPACKAGE:BOOL=OFF -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON"
 CMAKE_PREFIX_PATH = "/sl/usr:/sl:"
 CMAKE_INCLUDE_PATH = "/usr/src/secondlife/llqtwebkit/install/include:/usr/src/secondlife/viewers/snowglobe/snowglobe-1.x-svn/include:"
 CMAKE_LIBRARY_PATH = "/usr/src/secondlife/llqtwebkit/install/lib:"
 
You can echo a few environment variables to see if things look sane,
for example in my case:
<nowiki>
% echo $TOPPROJECT
/usr/src/secondlife/viewers
% echo $CXX
ccache g++
% echo $PROJECTBASE
/usr/src/secondlife/viewers/snowglobe-1.x-svn
% echo $REPOBASE
/usr/src/secondlife/viewers/snowglobe-1.x-svn/linden
% echo $INDRA
/usr/src/secondlife/viewers/snowglobe-1.x-svn/linden/indra
% echo $CONFIGURE_OPTS
--type=Debug -m64 --standalone
% echo $BUILDDIR
viewer-linux-x86_64-debug
</nowiki>
Note that latter two must match. Without the -m64 you'd have i686 (or whatever uname -m returns) instead
of x86_64, and without the --type=Debug you'd have '-relwithdebinfo' instead of '-debug'.
Now, independent of what viewer tree you are in, you can configure the viewer by typing:
 configure
This will run 'make configure', which executes the code defined in $ROOT/viewers/env.viewers.
Then, to compile and link the viewer, you can run:
 make
For the other targets, see $ROOT/viewers/env.viewers
All in all you will have to tweak and tune these files a bit for your system and liking,
but a MAJOR advantage of this system is that you only have to do that once, and then
it will work in the future without that you have to remember a lot. If you ever want
to change the behavior, then you just edit env.source or env.viewers.
Running the viewer no longer requires you to make a package. That is, you can
change env.source and set -DPACKAGE:BOOL=OFF. Simply cd into $BUILDDIR/newview
and run ./secondlife-bin from there: the configure command adds
symbolic links to everything that is needed to run it in-place, saving a lot
of time every run because the package doesn't have to be build anymore.
In that regard, note that a link was made to $TOPPROJECT/SLVoice. In my case (64 bit)
that is a little script that exectutes the 32-bit SLVoice process from one
of the tar balls provided by Linden Lab. You will have to fix this to make
voice work without packaging (put whatever you used there, or change the symbolic link).
=== Using quilt ===
The system also allows for using quilt transparently for every viewer directory.
In order to use it, run once for each viewer:
 cd $REPOBASE
 ln -s ../quilt/pc .pc
 quilt upgrade
The added 'quilt' function (in env.viewers) automatically adds '-p0' to a quilt refresh,
assuring that all your patches are consistently -p0, as is the output of 'svn diff'.
Of course, if you import a patch that is -p1 it will be -p1 until you run 'quilt refresh' once,
but for the rest you don't have to check anymore which patches are -p0 and which are -p1:
they are just all -p0.
The 'svn' function has been overloaded to take 'svn diff' and run 'quilt diff -p0 -z' instead,
if you have some quilt patch applied, which then prints the diff relative to the last time
you ran 'quilt refresh'. That way the 'quilt refresh' feels like a local commit and you can
use 'svn diff' to inspect uncommitted changes as usual.
'svn commit' has been overloaded to stop you from accidently using it when there are still
quilt patches applied, since then 'svn diff' doesn't output the REAL diff: a commit would
commit the whole applied quilt series!
I realize there is loads of room for improvement here, but this is what I current use.