uprobes: Move __set_bit(UPROBE_SKIP_SSTEP) into alloc_uprobe()

Cosmetic. __set_bit(UPROBE_SKIP_SSTEP) is the part of initialization,
it is not clear why it is set in insert_uprobe().

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
uprobes: Kill the "uprobe != NULL" check in uprobe_unregister()

Trivial. uprobe can't be NULL after mutex_unlock(), it was already used.

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
uprobes: Kill the pointless inode/uc checks in register/unregister

register/unregister verifies that inode/uc != NULL. For what?
This really looks like "hide the potential problem", the caller
should pass the valid data.

register() also checks uc->next == NULL, probably to prevent the
double-register but the caller can do other stupid/wrong things.
If we do this check, then we should document that uc->next should
be cleared before register() and add BUG_ON().

Also add the small comment about the i_size_read() check.

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
uprobes: Kill uprobe_consumer->filter()

uprobe_consumer->filter() is pointless in its current form, kill it.

We will add it back, but with the different signature/semantics. Perhaps
we will even re-introduce the callsite in handler_chain(), but not to
just skip uc->handler().

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
uprobes: Introduce filter_chain()

Add the new helper filter_chain(). Currently it is only placeholder,
the comment explains what is should do. We will change it later to
consult every consumer to decide whether we need to install the swbp.
Until then it works as if any consumer returns true, this matches the
current behavior.

Change install_breakpoint() to call filter_chain() instead of checking
uprobe->consumers != NULL. We obviously need this, and this equally
closes the race with _unregister().

Change remove_breakpoint() to call this helper too. Currently this is
pointless because remove_breakpoint() is only called when the last
consumer goes away, but we will change this.

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
uprobes: _unregister() should always do register_for_each_vma(false)

uprobe_unregister() removes the breakpoints only if the last consumer
goes away. To support the filtering it should do this every time, we
want to remove the breakpoints which nobody else want to keep.

Note: given that filter_chain() is not actually implemented, this patch
itself doesn't change the behaviour yet, register_for_each_vma(false)
is a heavy "nop" unless there are no more consumers.

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
uprobes: _register() should always do register_for_each_vma(true)

To support the filtering uprobe_register() should do
register_for_each_vma(true) every time the new consumer comes,
we need to install the previously nacked breakpoints.

Note:
	- uprobes_mutex[] should die, what is actually protects is
	  alloc_uprobe().

	- UPROBE_RUN_HANDLER should die too, obviously it can't work
	  unless uprobe has a single consumer. The consumer should
	  serialize with _register/_unregister itself. Or this flag
	  should live in uprobe_consumer->state.

	- Perhaps we can do some optimizations later. For example, if
	  filter_chain() never returns false uprobe can record this
	  fact and avoid the unnecessary register_for_each_vma().

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
uprobes: Introduce uprobe->register_rwsem

Introduce uprobe->register_rwsem. It is taken for writing around
__uprobe_register/unregister.

Change handler_chain() to use this sem rather than consumer_rwsem.

The main reason for this change is that we have the nasty problem
with mmap_sem/consumer_rwsem dependency. filter_chain() needs to
protect uprobe->consumers like handler_chain(), but they can not
use the same lock. filter_chain() can be called under ->mmap_sem
(currently this is always true), but we want to allow ->handler()
to play with the probed task's memory, and this needs ->mmap_sem.

Alternatively we could use srcu, but synchronize_srcu() is very
slow and ->register_rwsem allows us to do more. In particular, we
can teach handler_chain() to do remove_breakpoint() if this bp is
"nacked" by all consumers, we know that we can't race with the
new consumer which does uprobe_register().

See also the next patches. uprobes_mutex[] is almost ready to die.

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
uprobes: Change filter_chain() to iterate ->consumers list

Now that it safe to use ->consumer_rwsem under ->mmap_sem we can
almost finish the implementation of filter_chain(). It still lacks
the actual uc->filter(...) call but othewrwise it is ready, just
it pretends that ->filter() always returns true.

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
uprobes: Kill UPROBE_RUN_HANDLER flag

Simply remove UPROBE_RUN_HANDLER and the corresponding code.

It can only help if uprobe has a single consumer, and in fact
it is no longer needed after handler_chain() was changed to use
->register_rwsem, we simply can not race with uprobe_register().

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
uprobes: Kill uprobe->copy_mutex

Now that ->register_rwsem is safe under ->mmap_sem we can kill
->copy_mutex and abuse down_write(&uprobe->consumer_rwsem).

This makes prepare_uprobe() even more ugly, but we should kill
it anyway.

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
uprobes: Kill uprobe_events, use RB_EMPTY_ROOT() instead

uprobe_events counts the number of uprobes in uprobes_tree but
it is used as a boolean. We can use RB_EMPTY_ROOT() instead.

Probably no_uprobe_events() added by this patch can have more
callers, say, mmf_recalc_uprobes().

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
uprobes: Introduce uprobe_is_active()

The lifetime of uprobe->rb_node and uprobe->inode is not refcounted,
delete_uprobe() is called when we detect that uprobe has no consumers,
and it would be deadly wrong to do this twice.

Change delete_uprobe() to WARN() if it was already called. We use
RB_CLEAR_NODE() to mark uprobe "inactive", then RB_EMPTY_NODE() can
be used to detect this case.

RB_EMPTY_NODE() is not used directly, we add the trivial helper for
the next change.

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
uprobes: Kill uprobes_mutex[], separate alloc_uprobe() and __uprobe_register()

uprobe_register() and uprobe_unregister() are the only users of
mutex_lock(uprobes_hash(inode)), and the only reason why we can't
simply remove it is that we need to ensure that delete_uprobe() is
not possible after alloc_uprobe() and before consumer_add().

IOW, we need to ensure that when we take uprobe->register_rwsem
this uprobe is still valid and we didn't race with _unregister()
which called delete_uprobe() in between.

With this patch uprobe_register() simply checks uprobe_is_active()
and retries if it hits this very unlikely race. uprobes_mutex[] is
no longer needed and can be removed.

There is another reason for this change, prepare_uprobe() should be
folded into alloc_uprobe() and we do not want to hold the extra locks
around read_mapping_page/etc.

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
uretprobes: x86 arch specific code to hijack return address

These function hijack the return address, replaces it with a
"trampoline" a piece of code that contains a breakpoint instruction.

Signed-off-by: Anithra P Janakiraman <anithra@linux.vnet.ibm.com>
Signed-off-by: Anton Arapov <anton@redhat.com>
uretprobes: trampoline implementation

The first time a uprobe corresponding to a uretprobe is hit for a
process, a trampoline slot is obtained in the uprobes_xol_area and
initialized with a breakpoint instruction. This slot is subsequently
used by all uretprobes.

Signed-off-by: Anithra P Janakiraman <anithra@linux.vnet.ibm.com>
Signed-off-by: Anton Arapov <anton@redhat.com>
uretprobes: handler uretprobe entry

When a uretprobe is registered the corresponding uprobe that is created
is assigned a dummy consumer. When this uprobe is hit, the
uretprobe_handle_entry function is invoked instead of the handler. This
creates a uretprobe_instance, hijacks the return address and replaces it
with the trampoline if it has not already been done so.

Signed-off-by: Anithra P Janakiraman <anithra@linux.vnet.ibm.com>
Signed-off-by: Anton Arapov <anton@redhat.com>
uretprobes: invoke handlers and restore return address when trampoline is hit

The uretprobe handlers are invoked when the trampoline is hit, on
completion the trampoline is replaced with the saved return address and
the uretprobe instance deleted.

Signed-off-by: Anithra P Janakiraman <anithra@linux.vnet.ibm.com>
Signed-off-by: Anton Arapov <anton@redhat.com>
uretprobes: register, unregister uretprobes

A uretprobe is specified by a file:offset:handler. A uprobe is created
at the offset specified. When this uprobe is hit a uretprobe instance
would be created and a probe placed instead of the return address of the
current function. The handler specified is invoked before replacing the
return address. A uretprobe therefore places two probes, firstly a
uprobe at offset, whose handler in turn places the second probe at the
return address at task runtime.

Signed-off-by: Anithra P Janakiraman <anithra@linux.vnet.ibm.com>
Signed-off-by: Anton Arapov <anton@redhat.com>
uretprobes: use uretprobe instead of uprobe

  ssia

Signed-off-by: Anton Arapov <anton@redhat.com>