Add display of Users to task view panel

Also, fix the remove button being initially disabled.