Fix strangeness when using bogus filters on int:

Changes:
 * Make sure trace_start returns -1 for int: if the provided filter
   failed to compile, rather than falling through to using
   trace_apply_filter on the first packet (and failing). This is also
   more consistent with what pcapint: already does.
 * Check for -1 return value from trace_apply_filter, rather than
   treating -1 as a successful match.