Apply Oslo ModelBase to NeutronBase

Oslo.db has ModelBase which implements most of NeutronBase's code.
Therefore NeutronBase should inherit from ModelBase to keep things DRY.

Fixes bug #1171055

Change-Id: I1a08636a6aa225da8b5d43a0bbcb9b59c057df42
Improve lbaas haproxy plugin_driver test coverage

 - also removes unnecessary delete_health_monitor() from haproxy plugin_driver

Fixes bug 1191007

Change-Id: Ic9ec1f705a94a2e246d1a4985e72e9c58d48191d
Limit min<=max port check to TCP/UDP in secgroup rule

icmp_type and icmp_code are mapped to port_min_range and port_max_range
respectively. For ICMP there is no constraint between type and code.
Thus port range min<=max check should be enforced only for TCP and UDP.

Also makes sure that ICMP type/code are 0 to 255 (both inclusive).
Previously a value with 0 to 65535 were accepted for ICMP type/code.

Fixes bug 1197760
Fixes bug 1197769

Change-Id: I70aaf6e02fee461fa97dc254db906d9efa173669
Import Oslo's common rootwrap to Neutron

Use the common oslo-incubator rootwrap rather than maintain a
specific fork within Neutron.

- Migrated DnsmasqFilter use in dhcp.filters to the new EnvFilter
- Changed environment passing in ip_lib's netns.execute so that
  it can be properly matched using IpNetNsExecFilter + EnvFilter.
  It now calls "ip netns exec ns env A=B C=D command" instead of
  "A=B C=D ip netns exec ns command". Adjusted tests accordingly.

All the other changes are coming directly from the Oslo "rootwrap"
module sync.

Notes:
- Neutron locates its rootwrap.conf in etc/ rather than in etc/neutron
- Neutron maintains a specific bin/quantum-rootwrap-xen-dom0 which
  requires additional config in rootwrap.conf

Both behaviors were preserved in this commit, but this may need to be
addressed in the future to simplify future oslo-rootwrap updates.

Implements bp: quantum-common-rootwrap

Change-Id: I02879942a9d1169a71aa4d684c1b9ec109a6de32
Preserve packet:byte counts in iptables_manager.

Ported the nova iptables manager code to neutron, so that we
use iptables-save/restore with the -c flag to save/restore
the chains and rules with their packet:byte counts.  All other
changes were ported as well to keep the code as similar as
possible between the two, although they will be different as
I had to fix other bugs found during testing.

Updated tests accordingly to account for new calls and
input/output changes in formatting.

Changed iptables_firewall code to add iptables rules in the same
order that iptables-save will print them: source/dest, protocol,
sport, dport, target; else iptables_manager won't be able
to find them to preserve their [packet:byte] counts.
Tweaked other rules accordingly as necessary.

Fixed a bug introduced in an earlier version of this patch where
_modify_rules() sometimes wouldn't match an existing rule correctly
if not top=true.

Fixes bug 1125393

Change-Id: I858c552d8a7ae24f52f8e8daa05ac37026705773
Register agent schedulers options in one place

Currently agent schedulers options are defined in neutron.scheduler and
every plugin that supports scheduling should register them individually.
It is reasonable to define and register options in one place - agentschedulers_db.

Fixes NeutronManagerTestCase when running separately by importing missing config option

Fixes bug 1198904

Change-Id: Ia9c1ad3fb4b71796401ef5507f741f3cc024a4da
Improve DHCP agent performance

Previously when starting the dhcp agent the sync_state() process would
be extremely expensive as it would query quantum server for each network.
In order to improve performance a get_active_networks_info() was added
so this information could be retrieved in one query rather than doing a
query for each active network.

The second part of this patch optimizes the logic to avoid calling
get_dhcp_port(). Previously, this method was called once for each network
which makes a call to get_subnets() and get_ports() unnecessarily as
the dhcp agent can determine itself if it needs to update a port or create a
port for dhcp.

This patch also threads the inital sync process and maintains backwards
compatibility with the previous rpc api.

There was also a trivial change to the nvp_plugin where filters are assumed to
be a dict.

implements blueprint improve-dhcp-agent-performance

Change-Id: I3b631057f595250dad76516faa9b421789f60953
port-update fails when using SELECT FOR UPDATE lock

In Postgresql, it will throw the bellow exception when
using SELECT FOR UPDATE lock combined with outer join.
"SELECT FOR UPDATE/SHARE cannot be applied to the
 nullable side of an outer join"

The reason can refer http://www.postgresql.org/ \
message-id/21634.1160151923@sss.pgh.pa.us

for this issue, I don't think outer join is necessary，
so I change it to use inner join.

Fixes bug #1191653

Change-Id: Ifc9ecad91324ce28399431ea77fe0865b6d8e523
Add status description field for lbaas objects

Fixes bug 1166925

Change-Id: I44d936afd8cd531a786c09c38bbc4702737ce99c
Validate dscp value less that 64

Previously neutron would raise a 500 error if the dscp value was greater
than 63. This patch adds validation to neutron so that a clear error
message is returned if the value is greater than 63.

Fixes bug 1200026

Change-Id: I41a93661952669d112463608be907eede1901490
Ensure a name of packet filter is stored in NEC plugin

This patch fixes NEC Plugin to handle a name of packet filter and added
"name" column into PacketFilter table.

Fixes: bug #1198223
Change-Id: Ic191b2793653b41496922f2fa4be4f410d71dc64
Be compatible with oslo.config 1.2.0a3+

The private API of oslo.config changed, and the Cisco
options parsing code was using it.
Use an explicit MultiConfigParser instance for parsing
the config files.

Fixes LP Bug #1196084

Change-Id: I7ffcac3c295491fe9ba8abc7e98f33157a48c51b
Quiet down a large log file heavy hitter.

This line of code serialized the python structure for the device and throws it
at the log file.  The string is about 90K characters, isn't very useful and we
found that it takes a very large amount of space in the log file causing them
to grow very quickly and fill our log disk.  This change extracts the device
name (e.g.  "tapcdc07af9-8a") and uses that in the log message.

Change-Id: I240eb811445752407583f0fb322ce061acb293b9
Fixes: Bug #1200321
Fix issue with pip installing oslo.config-1.2.0

Fixes bug #1194807

Firstly, we update the oslo.config dep to 1.2.0a3 because of the issue
with namespace packages (bug #1194742).

But the main issue here is that if you currently do:

  $> pip install -r quantum/requirements.txt

then you end up with the oslo.config 1.1.1 code installed. This is
because oslo.config>=1.1.0 gets pulled in as a transitive dep and pip
gets confused. You can reproduce with e.g.

  $> pip install \
       http://.../oslo.config-1.2.0a3.tar.gz#egg=oslo.config-1.2.0a3 \
       python-keystoneclient
  $> pip freeze | grep oslo.config
  oslo.config-1.2.0a3
  $> python -c 'from oslo.config.cfg import DeprecatedOpt'
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  ImportError: cannot import name DeprecatedOpt

This is because of a bug with pip where it sees oslo.config-1.2.0a3 and
oslo.config as two unrelated things. It should strip the version part of
the egg= fragment before using it as a package name, but it doesn't.

However, we can simply use the -f/--find-links pip option in our
requirements.txt to add the tarball URL to the list of URLs considered
and also add the oslo.config>=1.2.0a3 dependency:

  $> pip install \
       -f http://.../oslo.config-1.2.0a3.tar.gz#egg=oslo.config-1.2.0a3 \
       'oslo.config>=1.2.0a3' \
       python-keystoneclient
  $> pip freeze | grep oslo.config
  oslo.config-1.2.0a3
  $> python -c 'from oslo.config.cfg import DeprecatedOpt'

This is actually exactly the semantics we want and we go to great
lengths in pbr to get these semantics while using a single tarball URL.
The only downside to this --find-links strategy is that we gain an extra
line in our requirements.txt ... but it does work around the pip bug.

Change-Id: I6f3eb5fd2c75615d9a1cae172aed859b36b27d4c
Merge "Validate dscp value less that 64"
Add option to ignore backend HTTP error in BigSwitch plugin

This patch fixes the way the BigSwitch plugin ignores 404
errors from the backend controller by allowing HTTP codes
to be selectively ignored on REST calls. This prevents
unnecessary exception handling and failover logic in calls
where 404 codes are acceptable.

Change-Id: Ia1118c039af3b45d96fb3f8a5fb5d3febdf50f4f
Fixes: bug #1200023
Merge "Be compatible with oslo.config 1.2.0a3+"
Merge "Quiet down a large log file heavy hitter."
Merge "Fix issue with pip installing oslo.config-1.2.0"
Refactor unit tests for NEC Plugin

blueprint nec-plugin-test-coverage

This commit refactors the base class of unit tests for NEC Plugin to
make writing new tests easier.

Add fake OFC Manager and change unit tests which use OFC Manager mock
to using this fake.  With this fake, we don't need to set return_value
and side_effect to the mock instance of OFC Manager in each test case.
This fake keeps all resources on OFC, and returns existence of
resources properly.  This fake is attached to OFC Manager mock instance
by setting all side_effects.  We can also set it to raise an exception.

Make NEC Plugin configurable in each test case.  Now we can configure
NEC Plugin by setting string in class value "_nec_ini" which will be
written into a temporal file and loaded by NEC plugin.

Move setup for callbacks into NecPluginV2TestCase() so that other unit
tests can use callbacks.

Change-Id: I3a52423bb5813547345677dc49abfea6563dd6cd