Ensure default route will exist after netconfig stage is done

Fixes: #PRD-1716

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
Put default route after interface creation

ISSUE: Centos affected only.
For system wide configured gateway (/etc/sysconfig/network), there is no ensure
it would has been applied as a default route. It will be applied only after node's
reboot or network restart.

SOLUTION:
We should avoid using "heavy" operations like 'service network restart', thus,
we should create default route by manually adding it into routing table while
its config stage (adding GATEWAY to /etc/sysconfig/network) is being executed as well.

We should also ensure right apply order as well (L3_if_downup <||> -> Defaultroute[$def_gateway])

Fixes: #PRD-1716

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
Forced vgreduce
Use ip route replace instead of route add

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
Ensure default route is purged before gateway-defined subint will be UP

ISSUE: Ubuntu affected only.
Assumes never put several interfaces to single L2 broadcast domain, if one of them is dhcp enabled, otherwise, for vlan/bond enabled interfaces, multiple or wrong default routes might have been created
on start-up time, thus, preventing desired gateway x.y.z.t would have been
added for default route on its static subinterface(vlan/bond slave) start-up.

F.e. for eth0 auto with dhcp and its eth0.100 subint with gateway x.y.z.t defined:
1) eth0 starts up
2) default route have created from dhcp-defined gateway f.g.h.i via eth0 (or ethZ, if it shares the
same L2 broadcast domain with eth0)
3) eth0.100 starts up
4) gateway x.y.z.t ignored as well ...

To avoid this, you can add pre-up hooks for every main/slave interface to erase existing
default route (if any) configured via this or any other main interface, but that would be an ugly solution.

Fixes: #PRD-2023

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
Add manual default route creation for Ubuntu case as well

ISSUE: Ubuntu affected only.
For interface with gateway defined (/etc/network/interfaces.d/ifcfg-fooX.YYY), there is no ensure
it would has been applied as a default route. It will be applied only after node's
reboot or network restart.

SOLUTION:
We should avoid using "heavy" operations like 'service network restart', thus,
we should as well create default route by manually adding it into routing table, after interface'
configuration and down&up operation has been completed.

We should also ensure right apply order as well (L3_if_downup <||> -> Defaultroute[$def_gateway])

Fixes: #PRD-2023
Depends on: #PRD-1716

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
Use ip route replace instead of route add

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
Revert "Ensure default route is purged before gateway-defined subint will be UP"

This reverts commit 74c87c740bb0c8bc32098c0bdffd284522fefbd3.
Fix network_setup checkbase array indexing

Missed indexing causes wrong startup order for interfaces.

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
Merge pull request #662 from bogdando/fix_prd2023

PRD-2023: Fix default route for Ubuntu
Merge pull request #641 from bogdando/fix_prd1716

PRD-1716: Fix default route for Centos
Merge pull request #686 from kozhukalov/ubuntu-disks-vgreduce-force

Forced vgreduce
place Ceph admin keyring in /etc/ceph

This change fixes Nova failure to attach a Ceph-backed volume to a VM.