It turns out that autobuild and the two contained commands all return 0 (success); at least when using "brew install -i mlkit" and "echo $?". I have no idea why the brew script terminates in non-interactive mode if we just have 'system "./autobuild"'. Here is a log:

new-host:local mael$ brew install --verbose --debug smltojs
/usr/local/Library/brew.rb: loading /usr/local/Library/Formula/smltojs.rb
/usr/local/Library/brew.rb: loading /usr/local/Library/Formula/autoconf.rb
/usr/local/Library/brew.rb: loading /usr/local/Library/Formula/mlton.rb
/usr/local/Library/brew.rb: loading /usr/local/Library/Formula/gmp.rb
/usr/local/Library/Formula/smltojs.rb: loading /usr/local/Library/Formula/autoconf.rb
/usr/local/Library/Formula/smltojs.rb: loading /usr/local/Library/Formula/mlton.rb
/usr/local/Library/Formula/smltojs.rb: loading /usr/local/Library/Formula/gmp.rb
/usr/local/Library/Formula/smltojs.rb: loading /usr/local/Library/Formula/python.rb
==> Downloading http://sourceforge.net/projects/mlkit/files/mlkit-4.3.7/mlkit-4.3.7.tgz
Already downloaded: /Library/Caches/Homebrew/smltojs-4.3.7.tgz
tar xf /Library/Caches/Homebrew/smltojs-4.3.7.tgz
==> ./autobuild
./autobuild 
/usr/local/Library/Homebrew/formula.rb:593:in `system'
BuildError: Failed executing: ./autobuild 
1. raise
2. ignore
3. backtrace
4. irb
5. shell
Choose an action: 1
==> Configuration
HOMEBREW_VERSION: 0.9.4
HEAD: f33f400446e5a10b67ec8d4585b47a3bf0339fc4
CPU: 8-core 64-bit ivybridge
OS X: 10.8.3-x86_64
Xcode: 4.6.2
CLT: 4.6.0.0.1.1365549073
X11: 2.7.4 => /opt/X11
==> ENV
HOMEBREW_CC: clang
MAKEFLAGS: -j8
CMAKE_PREFIX_PATH: /usr/local
CMAKE_INCLUDE_PATH: /usr/include/libxml2:/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers/
CMAKE_LIBRARY_PATH: /System/Library/Frameworks/OpenGL.framework/Versions/Current/Libraries
PKG_CONFIG_LIBDIR: /usr/lib/pkgconfig:/usr/local/Library/ENV/pkgconfig/10.8
ACLOCAL_PATH: /usr/local/share/aclocal
OBJC: cc
PATH: /usr/local/Library/ENV/4.3:/usr/local/opt/autoconf/bin:/usr/local/opt/mlton/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/texlive/2012/bin/x86_64-darwin

Error: smltojs did not build

new-host:local mael$ brew install -i smltojs
==> Downloading http://sourceforge.net/projects/mlkit/files/mlkit-4.3.7/mlkit-4.3.7.tgz
Already downloaded: /Library/Caches/Homebrew/smltojs-4.3.7.tgz
==> Entering interactive mode
Type `exit' to return and finalize the installation
Install to this prefix: /usr/local/Cellar/smltojs/4.3.7
bash-3.2$ ./autobuild
bash-3.2$ echo $?
0
bash-3.2$ exit
exit
Error: Empty installation
new-host:local mael$