davepacheco/mod_usdt — Issue #1: provider not automatically discovered by "dtrace -l"
Author: davepacheco
Created: 2011-12-02 17:54:47+00:00
State: open

Once apache is started, 'dtrace -l -n "httpd*:::"' reports no probes. If you specify particular pids though (as in 'dtrace -l -n "httpd12345:::"'), the probes show up. After that, they show up in the wildcard case as well. So specifying pids causes them to be auto-created, but USDT probes should always be auto-created.

This appears to be related to the way we're building mod_usdt.so. The Makefile says:

#
# The shared object combines the object files generated from C sources with the
# one generated by DTrace. XXX We use ld directly here rather than gcc because
# the Apache build process leaves the shared object with unreferenced symbols,
# which gcc doesn't like. Admittedly this is kind of weird, but it works
# because Apache will dlopen() the shared object from a context where these
# symbols are defined. ld allows this by defalt for shared libraries, but
# there should be some way to convince gcc that it's okay. Note that ld emits
# warnings here because it doesn't like being invoked outside a compiler
# driver.
#
$(SOFILE): $(OBJFILES) $(EXTRAOBJFILES)
$(LD) $(SOLDFLAGS) $(LDFLAGS) -o $@ $^

In other words, we're using ld to build the final shared object because of some gcc problem, and ld reports this warning:

dap@devel mod_usdt $ make
mkdir -p build
dtrace -xnolibs -h -o build/httpd_provider.h -s src/httpd_provider.d
gcc -Wall -Werror -fPIC -Ibuild -I/opt/local/include/apr-1 -I/opt/local/include -I/opt/local/include/db4 -D_REENTRANT -I/usr/include -DLDAP_DEPRECATED -I/opt/local/include/httpd -c -o build/usdt.o src/usdt.c
dtrace -xnolibs -G -o build/httpd_provider.o -s src/httpd_provider.d build/usdt.o
ld -shared -fPIC -o build/mod_usdt.so build/usdt.o build/httpd_provider.o
ld: warning: symbol '_init' not found, but .init section exists - possible link-edit without using the compiler driver
ld: warning: symbol '_fini' not found, but .fini section exists - possible link-edit without using the compiler driver

I had totally misread http://blogs.oracle.com/rie/entry/init_fini_not_found_use and thought these warnings were innocuous. However, when I change LD to GCC in the above recipe, the probes get created as expected. I didn't dig all the way to root cause, but this makes some sense in that we likely weren't running the dtrace DOF init routines when loading the library. (I'm not sure why these probes got autocreated with a more specific pid, but this also isn't totally surprising.)