Repository: netbox-community/netbox

Issue:
Race condition when POSTing to /api/ipam/prefixes/<id>/available-ips/
### Environment
* Python version: 3.5.3
* NetBox version:  2.4.6

### Steps to Reproduce
Create a normal prefix, then run this code against the API (replace `$token`, `$prefixid` and `$host`):

`for i in $(seq 16); do curl -X POST -so/dev/null -H "Authorization: Token $token" -H "Accept: application/json; indent=4" https://$host/api/ipam/prefixes/$prefixid/available-ips/ & done`

This code requests 16 IP addresses simultaneously from Netbox triggering the race condition bug.

### Expected Behavior
Netbox assigns IP addresses from .1 up to .16: [Example screenshot](http://xor.meo.ws/8b820a1a/ab06/4ecc/b226/260e6076c512.png)

### Observed Behavior
Netbox assigned IP addresses from .1 up to .5, assigning some of the IPs multiple times: [Example screenshot](http://xor.meo.ws/b993ccfc/632f/4372/b7b3/c0f2398e9ddb.png)


Discussion:
What's your setting for '[ENFORCE_GLOBAL_UNIQUE](https://netbox.readthedocs.io/en/latest/configuration/optional-settings/#enforce_global_unique)'?
@sdktr It was set to `False`. Setting it to `True` and restarting still allows the command above to create duplicate addresses. Just to confirm that the option was correctly set, manually creating a duplicate address in the web interface causes an error `Duplicate IP address found in global table: 100.64.8.1/22
`
I don't know that there's a feasible way to prevent this given the nature of the function. However, you can easily work around it by sending a single request for multiple addresses. For example, POSTing a list of five empty objects will return the first five available IPs.

```
curl -X POST \
-H "Authorization: Token $token" \
-H "Content-Type: application/json" \
-H "Accept: application/json; indent=4" \
https://$host/api/ipam/prefixes/$prefixid/available-ips/ \
--data '[{}, {}, {}, {}, {}]'
```

(Each object can optionally specify a VRF, which is why NetBox expects a list of objects rather than simply a count of IPs to create.)
@jeremystretch We have multiple VM hypervisors access phpIPAM to do our IP management. We wanted to give Netbox a try for DCIM and IPAM but duplicate IP assignments are a no-go.

Our workaround at the moment for this is to add a unique constraint to the ipam_ipaddress table:

```
ALTER TABLE ONLY ipam_ipaddress ADD CONSTRAINT ipam_ipaddress_ukey UNIQUE (vrf_id, address);
```

This makes Netbox throw an error in case of it trying to create duplicate IP entries. If we see an error related to the PSQL unique constraint or `{"non_field_errors":["Duplicate IP address found in VRF testing (testing): 100.64.8.3/22"]}` (which does happen sometimes when using VRFs), we back off for a few seconds and then retry.
I don't understand why this is closed? The API is used for automation, but cannot be used because this API is implemented incorrectly. What about the "nature of the function" prevents this? Why can't it be protected against as a key constraint, like what Fusl did above?

I understand you're just one person and there's only so much time, but how can anyone else consider making a pull request for issues that are just shut down, with the assertion that a fix is impossible? It's not always possible to just batch a request like that at once, you can't just change the way your automation works to account for a broken API. I'm trying to write a Terraform provider, where resources are handled individually, they cannot be batched into a single API call.
I believe this is the proper fix given how Django models work:
https://github.com/netbox-community/netbox/compare/develop...mattolenik:available-api-race

Can this be reopened so a PR can be submitted?
This was closed because no acceptable modification was proposed, and a workaround was provided.

> I believe this is the proper fix given how Django models work

This does not address the underlying issue:

```
prefix = Prefix.objects.select_for_update().get(pk=pk)
```

That locks _the parent prefix_ being queried; it does not prevent duplicate child prefixes or IP addresses from being created.
It doesn't prevent the insertion of another row with the same address, but it prevents this method from running concurrently. This most certainly does fix the problem with concurrent API calls. I can provide test cases that prove this, but not if it's just going to be shot down. I don't understand the attitude of not wanting to fix a serious race condition. This is a very serious bug and shouldn't be closed regardless of whether or not a fix was proposed...what you're telling people is that they can never attempt to fix this, because you'll never accept a fix for it.

Please, I'm literally trying to write an open source provider that people can use to help them use NetBox. Why shoot this down?
I see this as a critical issue.  The data model should never allow duplicate IPs to be inserted.

@jeremystretch Do you have a recommendation on an approach that would fix the issue more completely?  Can we also reopen this issue?

Full disclosure: I work with @mattolenik and this issue is a blocker for us being able to rollout Netbox as our central IPAM solution.  We want to do everything we can to avoid maintaining an internal fork and will gladly devote some dev time to fixing this properly.
@mattolenik @nicpar are one of you volunteering to own the fix as well as any follow-on issues from implementing the change? I am focused on v2.7 work for the near future and don't have any cycles for this. (Maybe one of the other maintainers can assist.)
@jeremystretch sure, I can do it! I'll start working on tests :)

Resolving patch:
diff --git a/netbox/ipam/api/views.py b/netbox/ipam/api/views.py
--- a/netbox/ipam/api/views.py
+++ b/netbox/ipam/api/views.py
@@ -1,6 +1,7 @@
 from django.conf import settings
 from django.db.models import Count
 from django.shortcuts import get_object_or_404
+from django_pglocks import advisory_lock
 from rest_framework import status
 from rest_framework.decorators import action
 from rest_framework.exceptions import PermissionDenied
@@ -10,6 +11,7 @@
 from ipam import filters
 from ipam.models import Aggregate, IPAddress, Prefix, RIR, Role, Service, VLAN, VLANGroup, VRF
 from utilities.api import FieldChoicesViewSet, ModelViewSet
+from utilities.constants import ADVISORY_LOCK_KEYS
 from utilities.utils import get_subquery
 from . import serializers
 
@@ -86,9 +88,13 @@ class PrefixViewSet(CustomFieldModelViewSet):
     filterset_class = filters.PrefixFilterSet
 
     @action(detail=True, url_path='available-prefixes', methods=['get', 'post'])
+    @advisory_lock(ADVISORY_LOCK_KEYS['available-prefixes'])
     def available_prefixes(self, request, pk=None):
         """
         A convenience method for returning available child prefixes within a parent.
+
+        The advisory lock decorator uses a PostgreSQL advisory lock to prevent this API from being
+        invoked in parallel, which results in a race condition where multiple insertions can occur.
         """
         prefix = get_object_or_404(Prefix, pk=pk)
         available_prefixes = prefix.get_available_prefixes()
@@ -180,11 +186,15 @@ def available_prefixes(self, request, pk=None):
             return Response(serializer.data)
 
     @action(detail=True, url_path='available-ips', methods=['get', 'post'])
+    @advisory_lock(ADVISORY_LOCK_KEYS['available-ips'])
     def available_ips(self, request, pk=None):
         """
         A convenience method for returning available IP addresses within a prefix. By default, the number of IPs
         returned will be equivalent to PAGINATE_COUNT. An arbitrary limit (up to MAX_PAGE_SIZE, if set) may be passed,
         however results will not be paginated.
+
+        The advisory lock decorator uses a PostgreSQL advisory lock to prevent this API from being
+        invoked in parallel, which results in a race condition where multiple insertions can occur.
         """
         prefix = get_object_or_404(Prefix, pk=pk)
 
diff --git a/netbox/utilities/constants.py b/netbox/utilities/constants.py
--- a/netbox/utilities/constants.py
+++ b/netbox/utilities/constants.py
@@ -27,3 +27,14 @@
     ('111111', 'Black'),
     ('ffffff', 'White'),
 )
+
+# Keys for PostgreSQL advisory locks. These are arbitrary bigints used by
+# the advisory_lock contextmanager. When a lock is acquired,
+# one of these keys will be used to identify said lock.
+#
+# When adding a new key, pick something arbitrary and unique so
+# that it is easily searchable in query logs.
+ADVISORY_LOCK_KEYS = {
+    'available-prefixes': 100100,
+    'available-ips': 100200,
+}


Test patch: