Fix shared_library_error() with the private loader

With the private loader, dlerror() will not give us information about
the error loading the library.  Usually it will return NULL, which would
crash in strncpy().

Our loader doesn't store the load error anywhere, so we can't return
anything informative.  We can however avoid crashing and let the user
know the load failed.

Review URL: https://codereview.appspot.com/6855106

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1748 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
add redirect functions for DrM i#1066
- add redirect functions: NtCreateFile, ZwCreateFile and many others
- add corresponding nt_raw_* functions
- replace DR calls to those functions with nt_raw_*

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1749 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
i#993: Fix off by one bug in bb->checked_end comparison

bb->checked_end is part of an open interval, so we need to use >= to
compare it against new_pc.  Adds a test case, selfmod_last_byte(), that
creates an app bb whose last byte is on a sandboxed page.

Fixes issue 993.

Review URL: https://codereview.appspot.com/6860043

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1750 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
i#989: Treat fallthroughs as transfers in check_new_page_contig()

This prevents building bbs that fall through from a readonly vmarea into
a sandboxed area, as well as other incompatible area transitions.  If
the first instr of the bb is straddling a vmarea, we let bb building
continue with the new vmarea flags so we don't create an empty bb.  This
case is covered by the selfmod_last_byte() test.

Fixes issue 989.

Review URL: https://codereview.appspot.com/6843126

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1751 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
i#994: Skip meta instrs and set correct translation for sandboxing checks

This fixes a failure to translate state for the fault in
security-common.selfmod while using the inscount sample
client.

Fixes issue 994.

Review URL: https://codereview.appspot.com/6871062

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1752 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
Fixed issue 995
- skip bb event callback for bb with invalid instructions

Review URL: https://codereview.appspot.com/6880043/

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1753 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
To support an including project exporting its own targets we need to export
our targets and our exports file but not the rest of our install files.  We
also want the includer to be able to send our binaries somewhere outside of
the install tree.  This adds two new variables to accomplish that: a bool
DO_DR_INSTALL_TARGETS and a path DR_INSTALL_TARGETS_DEST.

Xref http://code.google.com/p/drmemory/issues/detail?id=1061

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1754 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
fixes issue 1007

Support nested init/exit calls in all extensions via a refcount that is
atomically adjusted using a new DR export: dr_atomic_add32_return_sum().

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1755 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
i#1010: Clean up after vfork+execve children before clone

For cases where the vforked child dies before reaching exec and fails to
clean up, we also tolerate stale fds being present in the fd_table.

Fixes issue 1010.

Review URL: https://codereview.appspot.com/6883050

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1756 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
Add further documentation on clean call optimization.
Add forgotten release note for i#1007 dr_atomic_add32_return_sum().

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1757 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
Honor non-empty CMAKE_BUILD_TYPE as controlling DEBUG var.
This includes not only the initial default but also changing the value.

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1758 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
fixes issue 997

Fix error in mangle_trace() assert on jump instr translation.

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1759 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
Don't add -O2 to clients in DynamoRIOConfig.cmake

Changed CLIENT_DEBUG to OPTIMIZE_SAMPLES, so we have a way of independently turning on optimizations for samples.

This fixes the drmemory build by silencing a warning that's only present
with -O2, but not -O3, which is what drmemory builds with in release
build.

Review URL: https://codereview.appspot.com/6902050

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1760 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
Consolidate find_dynamo_library() into tools.c for Linux tests

Review URL: https://codereview.appspot.com/6897047

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1761 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
i#1008: Remove aggressive restorer_valid[i] == -1 assert

If we've ever received a signal without an SA_RESTORER flag and with a
non-NULL sa_restorer value, then this assert will trigger when creating
a child thread.  Removing it doesn't appear to break any invariants.

I re-ordered the memset with the installation of our signal info field
to avoid a possible race on fully_initialized, which was for i#26.

Fixes issue 1008.

Review URL: https://codereview.appspot.com/6904055

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1762 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
Handle erroneous double calls to exit.
Use unique names for easier debugging when combining static ext libs.

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1763 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
fixed issue 1011
- split NT_SYSCALL to NT_RAW_SYSCALL to avoid static link
- put most nt_raw_syscall into DynamoRIO CORE

Review URL: https://codereview.appspot.com/6923053/

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1764 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
Fix missing support for VS generator in .drpath files

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1765 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
Add a -hashtable_study option to allow disabling hashtable studies

Large apps tend to trigger hashtable curiosities and generate large
blocks of hashtable entry logging that obscure interesting assertions
and logs.

Review URL: https://codereview.appspot.com/6925048

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1766 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6
i#1012: Don't uninstall DR's signal handlers while cleaning up other threads

Fixes issue 1012.

Review URL: https://codereview.appspot.com/6912053

git-svn-id: https://dynamorio.googlecode.com/svn/trunk@1767 49cc7528-f6fd-11dd-9d1a-b59b2e1864b6