squashed merge of ccng-side-by-side to master

cloud_controller_ng shortlog c012225d..c810a010b

Jesse Zhang (50):
      MessageBus.subscribe with inbox
      Ignore yardoc rbenv and OS X metadata files
      Merge "MessageBus.subscribe with inbox"
      Legacy bulk api support
      Merge "Legacy bulk api support"
      custom json matcher
      Start and stop instances
      respond_with matcher
      Process Health Manager requests
      Minor fixes to doc / comments
      Validate state and package_state for an app
      update_uris for dea client
      Deserialize environment json on attribute read
      Mark for staging after touching app
      mark app for restage when BUNDLE_WITHOUT changes
      Refactor legacy app request translation
      Separate update and create request translations
      Fix typo in test case name
      Merge changes Idc3a26f0,Ie706a708,I36da969b,I0baf2c1a,I7ed32223,I423aebf2
      Merge changes I997e0c70,I7402710c,Ic89990aa
      Legacy apps env support
      Mandatory bulk_api in config
      Add test for cc_partition
      Merge "Legacy apps env support"
      Add test for legacy apps api read
      Decode UAA tokens when it looks like one
      [fix] Health Manager stop request
      Ensure app is staged before talking to DEA
      respond_with RSpec Matcher
      Merge changes I6092f7e2,Icf383aac
      [fix] Stub out app stager before each model test
      after_modify hook for model api
      dea_update_pending? on app objects
      Update DEA when an app's associated routes change
      Update DEA when a route's associated apps change
      nginx upload offloading
      nginx X-Accel for staged droplet download
      nginx offloading
      distinct end points for droplet download / upload
      Merge changes I95c91f7e,I30cde51d
      [tests] composable config-override
      Return 501 when gateways do not support lifecycle
      Use old files url when serving files from CC
      Properly encode tail param in query string
      Use local bind address
      World readable runtimes, frameworks and services
      serialization data service API support
      Require presence of auth token for gateways
      Authenticate requests to list_handles
      [fix] Fix list_handles authentication

Kowshik Prakasam (65):
      Implemented MessageBus#request.
      Added DeaClient.dea_request, .dea_publish methods.
      Implement DeaClient.find_specific_instance.
      Implement DeaClient.get_file_url
      Use '/' in exceptions when path is nil.
      Implement Files API.
      Add files wrapper for legacy apps.
      Add cc_partition flag to DEA start messages.
      Add timeout option to MessageBus.request.
      Implement DeaClient.find_instances.
      Minor styling fix.
      Define StatsError.
      Implement DeaClient.find_stats.
      Implement stats API.
      Styling fixes.
      Add stats wrapper for legacy apps.
      Improve test description.
      Clean up unit tests by using mock DeaClient.
      Check indices of instances in NATS response.
      Define class HealthManagerClient.
      DeaClient.find_instances allows MessageBus options
      Define InstancesError.
      Implement DeaClient.find_all_instances
      Implement V2 'instances' REST API endpoint.
      Add 'instances' wrapper for legacy apps.
      Fix styling.
      In files API, send path to DEA via NATS.
      Replace credentials with tuple.
      Fix styling.
      Add LegacyApiBase#has_default_space? method.
      Add helper to create user without default space.
      Don't return account usage if no default space.
      Fix typo.
      Add D100 filter for /info/services end-point.
      Prep files API for file server redirection.
      Use default values for credentials.
      Replace "url" with "uri".
      s/v1_api/redirect_ok.
      Remove obsolete comment.
      Fix bug that causes subscription on nil.
      Invert if block condition for better readability.
      In files API, redirect to file uri v2.
      Re-order tests to improve readability.
      Implement HealthManagerClient.find_crashes
      Implement crashes API.
      Add 'crashes' wrapper for legacy apps.
      Merge "Implement HealthManagerClient.find_crashes"
      Merge "Implement crashes API."
      Merge "Add 'crashes' wrapper for legacy apps."
      Refactor files API to improve readability.
      Catch HTTP 416 response from dea file server.
      Flag to override stopped state for app stats.
      Refactor v2 stats API to allow app stopped state.
      Allow stopped state in legacy stats API.
      Send email to service gateway during provision.
      Clear request guid in steno logging after request.
      Fix crashes API spec.
      Merge "Fix crashes API spec."
      Add play framework.
      Merge "Add play framework."
      Bump steno to include fix for ephemeral IO errors.
      NATS timeout in DeaClient.find_specific_instance
      Don't subscribe when no messages are expected.
      Publish droplet.updated when an app is updated.
      Make healthy_instances call work for multiple apps

Patrick Bozeman (220):
      remove service_id from service_instance
      accept empty attributes in an api message
      disallow space dev to update service binding
      provision/deprovision service instance with gw
      split label and version for legacy gw handle api
      bind and unbind on sg
      throw correct domain errors
      clarify spec desc for enumeration with perms
      refactored org permission checks
      remove duplicated method
      permissions for domains
      update new cc <-> gw message format
      require hm respondent
      pass mbus subscribe options down to nats
      v2 route api
      import/export routes in app
      Models::App uris method
      allow setting guid relations on unsaved objects
      legacy app api uri support
      send correct uris to dea on app start
      allow admins to create shared domains
      move migrator out of initial migration
      Model::Domain.default_serving_domain
      add default serving domain to org on create
      inherit shared domains from org on space creation
      switch legacy apps spec to use default domain
      add explicit org ownership to a route
      stronger relation validation for routes
      better translation of errors for InvalidRelations
      lookup route via visible routes in legacy apps
      fix env encoding for legacy app api
      send updated_at in hm bulk api
      add version to bulk hm api export
      add version check to hm bulk api spec
      correctly parse hm bulk api resume id
      use partition in hm subscription
      save app after updating droplet hash from stager
      correct encoding of id and token in bulk api
      fix app key_changed?
      send env as array with = to dea and stager
      Merge "send env as array with = to dea and stager"
      fix spec desc typo
      send quota event on service binding
      send quota calls on domains
      send quota events for routes
      hm client healthy_instances
      return running instances via legacy api
      fix message format of hm flapping status call
      yeti setup script
      :case_insensitive support for sequel Strings
      turn down debug logs for specs
      convert named things to be case insensitive
      Merge "turn down debug logs for specs"
      Merge "convert named things to be case insensitive"
      fix TEAM-59: add node08 to config
      fix TEAM-58: add legacy service offering endpoint
      Merge "fix TEAM-59: add node08 to config"
      fix TEAM-60: send 404 instead of 400
      fix team-61: allow model/stack during app create
      explict test for exception processing on the mbus
      pass staging client errors correctly back to ccng
      stop apps on deletion
      Merge "explict test for exception processing on the mbus"
      Merge "pass staging client errors correctly back to ccng"
      Merge "stop apps on deletion"
      Merge "[fix] Health Manager stop request"
      fix stopping instances vs indices in dea client
      send svc provider and vendor to dea on app start
      re-enable state on app create
      bump em to 1.0.0
      ruby 1.9.3 compatability changes
      AppStger.delete_droplet
      delete staged droplets on app deletion
      send old style svc label-version to stager
      include uaa token in security context
      legacy users api
      legacy get /services/:name
      support creation of N users for parallel yeti.
      add metadata to app model
      add command= to app model
      import command for apps and export metadata
      send app metadata to stager
      legacy app api stand alone command support
      remove metadata from v2 api
      add console support to app model
      add console support to dea client start msg
      add console support to app api
      range support for non-nginx version of files api
      Merge "range support for non-nginx version of files api"
      comment out dea v2 file url support
      get console support check correct
      Merge "comment out dea v2 file url support"
      Merge "get console support check correct"
      AppPackage.delete_package
      remove gratuitous stub in specs
      Merge "AppPackage.delete_package"
      Merge "remove gratuitous stub in specs"
      delete app packages on app deletion
      extra debug logging in hm respondent
      extra logging in resource pool
      don't leak exceptions into em defer
      Merge "nginx upload offloading"
      Merge "nginx X-Accel for staged droplet download"
      bump vcap_common
      Revert "nginx upload offloading"
      Revert "nginx X-Accel for staged droplet download"
      add basic varz for our sinatra apis
      Merge "s/v1_api/redirect_ok."
      Merge "Revert "nginx upload offloading""
      Merge "Revert "nginx X-Accel for staged droplet download""
      Merge "bump vcap_common"
      Merge "add basic varz for our sinatra apis"
      Merge "Fix bug that causes subscription on nil."
      basic varz support for sequel
      turn on varz for sequel
      fix db setup bug
      change sample config to default to localhost
      Merge "change sample config to default to localhost"
      don't abandon a qm token if we failed to commit it
      add vcap request id to thread local storage
      send vcap request id to qm during quota calls
      tell sequel gem to keep name, etc as strings
      send service config to gw when it enumerates handles
      Merge "tell sequel gem to keep name, etc as strings"
      Merge "send service config to gw when it enumerates handles"
      migrate svc instance and binding data to json
      fix/finalize legacy crashes api
      Merge "migrate svc instance and binding data to json"
      Merge "fix/finalize legacy crashes api"
      bump steno
      vcap ring buffer
      return ringbuffer or recent errors in varz
      extra debugging in base model controller
      optimize serialized relations_hash
      refactor spec env setup into a helper class
      speed up model specs
      move spec setting of quota mgr to always be blind
      remove reset_db from common api tests
      remove reset_db from permission tests
      remove reset_database from legacy_apps_spec
      remove reset_db from legacy_bulk before :each
      remove reset_db from legacy info
      speedup collection tests
      misc speedups for api unit tests
      dea client spec speedup
      minor timestamp spec speedup
      improved domain validation regex
      remove old template_obj from model specs
      make model deserialization spec helper more robust
      guard config for ccng
      add wildcard flag to domains
      calculate intermediate domains for a domain name
      overlapping domain validation checks
      keep relation depth in next/prev pagination urls
      use caller supplied request guid if it exists
      always paginate in a known order
      return error code 100 (in body) on bad auth token
      wildcard domain support for routes
      domain wildcard removal policy
      white space fixup
      fix domain permission check for system domains
      fix inverted wildcard logic
      refactor shared domain creation
      support multiple serving domains
      add shared_domains subset
      don't allow using the shared domains directly
      use destroy in specs to allow sequel hooks to run
      shared domain inheritance and visibility for orgs
      fix legacy route mapping
      Merge "Updated set_yeti to work with version UAA client 1.2.4"
      nil host route spec fixup
      move route ownership from org to space
      fix space domain test
      fix domain spec typo
      minor reorg of domain model spec
      add more domain auto-association tests
      change instance of spec delete to destroy
      better cleanup in collection specs and use shared domains in org spec
      auto-associate owned domains with their org
      Merge branch 'custom_domains'
      fix route migration for existing routes
      add billing flag to org
      optionaly disable model tests
      only run uniqueness tests if needed
      be explict with model requires
      initial billing event model
      make object serialization configurable
      initial billing events api
      remove old quota manager calls
      alias org billing_enabled?
      AppStartEvent.create_from_app
      AppStopEvent.create_from_app
      OrganizationStartEvent.create_from_org
      ServiceCreateEvent.create_from_service_instance
      ServiceDeleteEvent.create_from_service_instance
      emit app start/stop billing events
      emit service create/delete events
      emit org billing start events
      add required start/end date params to billing api
      s/id/guid for billing events
      org summary endpoint
      space summary endpoint
      app summary endpoint
      remove bad export on service plan
      Health and framework/runtime info for space summary
      add health to app summary
      fix return from healthy_instances
      enhance space space summary
      add new style runtime config file support
      add new style framework config file support
      switch to loading new format rt and fw
      remove old legacy frameworks config
      refactor DeaClient.get_file_uri
      don't expose v1 dea uri on a fetch failure
      extract get_file_uri_for_search_param in files api
      s/instance_id/instance in files_spec
      DeaClient.get_files_for_instance_id
      add instance_id search param to files api
      send fw and rt info in staging messages
      send fw info in dea messages

Ryan Tang (1):
      Updated set_yeti to work with version UAA client 1.2.4

dea_next shortlog ed1e551..1362250

Anfernee Gui (2):
      Generate private id for running instance
      Merge "Generate private id for running instance"

Kowshik Prakasam (33):
      Move deaClient and common code to separate files.
      Remove unwanted symbol exports.
      Ignore write errors when client has disconnected.
      Move file streaming API to separate file.
      Do case sensitive check on tail query parameter.
      Add comments.
      Re-order method definition for better readability.
      Merge "Move deaClient and common code to separate files."
      Merge "Remove unwanted symbol exports."
      Merge "Ignore write errors when client has disconnected."
      Merge "Move file streaming API to separate file."
      Merge "Do case sensitive check on tail query parameter."
      Merge "Add comments."
      Merge "Re-order method definition for better readability."
      Dont use pointers for slices.
      Change application_id from Integer to String.
      Deprecate application_users but preserve it in env
      Default value for path when generating file uri v2.
      Pass empty tags in dir server registration.
      Revert "Lazily fill runtimes hash"
      Use line comments instead of block comments.
      Fix comment.
      Access type http.Header through its API.
      Refactor to prepare code for byte range queries.
      Close file handles.
      Implement handler for http range queries.
      Better err message when runner cant read config.
      Improve comments.
      Merge branch 'master' into ccng-integration
      Fix seek constant.
      Merge branch 'master' into revert_runtime_changes
      Revert "Runtime messages now use the version_output attribute"
      Merge branch 'master' into revert_runtime_changes

Patrick Bozeman (1):
      Merge branch 'master' into ccng-integration

Pieter Noordhuis (38):
      Update vcap_common
      Update steno
      Remove unused code
      Use real and absolute path to tmpdir
      Add cc_partition attribute to instance
      Explicitly stringify environment pairs
      Update service schema
      Handle nils in runtime environments
      Runtime messages now use the version_output attribute
      Stringify droplet attribute for incoming messages
      Merge "Runtime messages now use the version_output attribute"
      Merge "Stringify droplet attribute for incoming messages"
      Stringify droplet attribute in more places
      Undo 75937ab0, fall back on 'instance_ids'
      Merge branch 'master' into ccng-integration
      Fix crash reaping
      Remove unused dependency
      Change crash reaper specs to hit the disk
      Schedule callback after reaping crash from thread
      Reap crashes when disk gets full
      Merge branch 'master' into ccng-integration
      Revert "Merge branch 'master' into ccng-integration"
      Update em-warden-client
      Close warden connections after destroy
      Load snapshot before connecting to NATS
      Compatibility with WIP version of warden
      Update eventmachine to 1.0.0 and use epoll
      Write spec logs to stderr
      Add coding directive
      Update em-http-request to 1.0.3
      Behavior of EM::Connection#error? changed
      Don't attempt to connect when health check is done
      Include reason in log message when crash is reaped
      Create crashes path at start time
      Update warden dependencies
      Update steno dependency
      Pass resource limits directly to warden
      Move generic warden functionality to base class

warden shortlog d8b0a93d6..b5ab5fc24

Bob Nugmanov (2):
      Whitelist safe devices
      Merge "Whitelist safe devices"

Kowshik Prakasam (9):
      Create work path before specs start.
      Resource limits for run and spawn commands.
      Plumb resource limits from warden server to wsh.
      Specify optional handle when creating a container.
      Introduce container id.
      Custom handle for create command.
      Return list of alive job ids in info response.
      Populate info response with list of alive job ids.
      Don't special case iomux-link exit status.

Pieter Noordhuis (77):
      First stab at warden shell
      Extract client_loop from client_accept
      Use CLOEXEC on pipes
      Dynamically allocate buffer space
      Enhance pump code
      Move hooks
      Change to right directory
      Remove unnecessary interpolation syntax
      Assert number of fds to receive
      Use extra fd for exit status
      Configure network interfaces from hooks
      Force link creation
      Update start/stop to use warden shell
      Mount /proc from hook
      Remove SSH related setup code
      Remove overrides now that upstart is no longer used
      Trim down setup.sh
      Move common.sh to lib/
      Change mount paths; include config from common.sh
      Ignore wshc and wshd
      Select for reads, block for writes
      Forward return value
      Skip fd extraction on EOF
      Skip fd extraction when pointer is not specified
      Add copy in case splice cannot be used
      Support interactive terminals
      Define buffer size in msg.h
      Fix number of fds to send to client
      Specify environment on exec
      Extract fork to separate function
      Forward argv to wshd
      Forward resource limits to wshd
      Remove unused files
      Rename wshc -> wsh
      Configurable process title for wshd
      Add run directory to skeleton
      Remove no longer required code from setup.sh
      Add option parsing to wsh to discard rsh options
      Update destroy script
      Fix fcntl error (F_SETFD vs F_SETFL)
      Set FD_CLOEXEC on client connection
      Undefine macro that is only used once
      Loop waitpid in wshd
      Forward user from RSH compatibility mode to wshd
      Add route for default gateway from hook
      Make stop script graceful again
      Fix regexp for matching tc output
      Modify server code to use wsh
      Remove the RLIMIT_RTTIME resource limit
      Copy wshd and wsh from setup task
      Break from waitpid loop when it -1 or 0
      Use -l flag to su to chdir to HOME
      Loop to write entire buffer
      Merge branch 'master' into wsh2
      Use /tmp/warden/spec as working directory for specs
      Use small buffer on stack instead of heap
      Change temporary directory used in container code
      Change user via wsh/wshd
      Fix setup task
      Use perror
      Setup rudimentary environment from wshd
      Pass logger to deferred child to preserve handle
      Use external utility to close non-default fds
      Update em-posix-spawn
      Merge branch 'master' into wsh2
      Update dependencies
      Combine resource limits in configuration and requests
      Remove unused code
      Close in/out/err in wshd when entering accept loop
      Merge remote-tracking branch 'origin/wsh2' into wsh2
      Merge branch 'wsh2'
      RLIMIT_AS must be specified in bytes
      Configure write side of pump pair to be blocking
      Merge "Configure write side of pump pair to be blocking"
      Remove dependency on vcap-common
      Update eventmachine to 1.0.0
      Whitelist ptty related devices

Yang Yang (1):
      make setquota optional

Change-Id: I2ebb9fae3fa7b16e1698a84b2edde5436c093350
Merge "squashed merge of ccng-side-by-side to master"