Repository: joke2k/faker

Issue:
Public IP blocks used with ipv4_public encompass reserved address namespaces
The `ipv4_public` faker generates IP addresses from public blocks, excluding private blocks configured in `IPV4_PUBLIC_NETS` and `IPV4_PRIVATE_NET_BLOCKS`, however it often results in addresses coming from reserved spaces.

One example is 192.0.0.145, or in general 192.0.0.0/24 addresses from reserved space, per https://en.wikipedia.org/wiki/Reserved_IP_addresses

This clashes with many popular libraries checking whether IP is public and routable such as `django-ipware`.

### Steps to reproduce

```
>>> startswith = False
>>> while not startswith:
...   ip = fake.ipv4_public()
...   if ip.startswith('192.0.0'):
...     startswith = True
...     print(ip)
...
192.0.0.145
```

### Expected behavior

No address starting with `192.0.0` generated

### Actual behavior

`192.0.0.145` comes up often.



Discussion:
Thanks for the report @maticomp :)

It will be a little while before I get the chance to work on this, but I'll be happy to review any PR :)
@fcurella thanks :) I'll try to post one soon, either by myself or with help of the devs working with me on the project leveraging the convenience of the faker :) Cheers!
@maticomp can you see what netmask is returning? I think the parameter is network=true. 

Nvm, looked at the code again. Looks like it generates some mask after the IP. Although, I noticed the ipaddress library is imported. Might be able to use that to get a valid ip address.
After reviewing the code for faker and the ipaddress code, i feel like we could definitely improve on this.

In faker, there are 3 classes defined, but there are also reserved ip addresses in there. I feel like we should remove those and go with the constants defined in the ipaddress class. https://github.com/python/cpython/blob/d5f017bbd65f37ac53fd3c6c439a53155eef2475/Lib/ipaddress.py#L1559

I believe this can be coded in a way using the ipaddress class and what it already offers and to quickly get a random ip address that is valid in any scenario. It's going to take a lot of refactoring, but should be worth it.

I'll try put together some kind of proof of concept when time permits and maybe make a determination from there.
@kungfu71186 while I agree that refactoring towards using what Python gives us in `ipaddress` is a great idea, I'm now even more confused! Things are getting interesting quickly 😄 

```
>>> import ipaddress
>>> i = ipaddress.ip_address('192.0.0.145')
>>> i.is_private
False
>>> i.is_global
True
>>> i.is_reserved
False
```

if you see the code you have quoted, it defines:

```
    _private_networks = [
        (...)
        IPv4Network('192.0.0.0/29'),
```

but RFC6890 specifies a full `/24` prefix as reserved for IANA.

https://tools.ietf.org/html/rfc6890#section-2.1

The full list of reserved address spaces is also hosted on IANA website:

https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml

The RFC is fairly fresh, but the full prefix was already assigned to IANA in https://tools.ietf.org/html/rfc5735 (2010) and https://tools.ietf.org/html/rfc3330 (2003).

I am now not sure why `ipaddress` - consistently with `faker`, one must say - considers only `/29` as private, while returning the rest from the full `/24` prefix as global and routable, which is not true.

Now I'm lost. 😄 
@maticomp I think it's actually a "bug". Needs to be updated.

https://docs.python.org/3/library/ipaddress.html#ipaddress.IPv4Address.is_private which does reference that same registry. 

This is also why i suggest we just use the official library and we can fix bugs in there as well :)

Would you be able to make pull request to fix all of them? Looks like there are few others missing as well.
@kungfu71186 I will try early next week, but I'm not certain I'm knowledgable enough with those special allocations not to make a mistake of my own.

The docs versus code of Python's ipaddress code is not entirely clear to me. 

E.g.

> is_global
> True if the address is allocated for public networks. See iana-ipv4-special-registry (for IPv4) or iana-ipv6-special-registry (for IPv6).

But the code says:

```
(...)
_public_network = IPv4Network('100.64.0.0/10')

```

which is a slightly confusing field name, which is later used as such:

```
(...)
def is_global(self):
    return self not in self._constants._public_network and not self.is_private
```

From the IANA table 100.64.0.0/10, a shared address space for carrier NAT services, has the same combination of global & forwardable flags as few other address spaces. It's public, but not global - but it's not the only one. I don't understand why it's treated specially, and there is a class field with a single network and not a list of such networks. 

I'm not really certain about the division between public globals, public non-globals, private and reserved spaces and the division is not thoroughly defined in `ipaddress` either.

What I can surely do is I can fairly quickly post a PR to update the `_private_networks` constant, but I'm not sure it's the only thing that should be done -- and Python contributing guidelines are not something I'm acquainted right now. I think an issue should be created on their bug tracker first either way, so the PR can refer ta BPO number can be referenced.

Maybe I'll start with a lightwieght PR bringing the `faker` constants up to speed, before refactoring `faker` to use the Python's library?
https://bugs.python.org/issue20857 that was over 4 years ago, lol. Probably won't get fixed unless a pull request is created.
@kungfu71186 yes, I've seen that ;-) 

On the other hand - since **faker**'s purpose is to generate viable randomized test data, do we NEED to use external libraries, and have a very broad scope of generated IPs?

Right now faker has a hardcoded list of private and reserved subnets (which is incomplete). It generates private IPs from those subnets, which is easy. However, public IPs are generated from public subnets partially overlapping with the private ones, after excluding private ones from the set.

The logic of that is quite convoluted and I wonder if it serves purpose. 

Wouldn't it be viable to have a hardcoded list of non-overlapping subnets for private and public ranges?

Let's say, private subnets:

`192.168.0.0/24, 10.0.0.0/8`

and public subnets:

`5.0.0.0/8, 8.0.0.0/8, 193.0.0.0/8`

and simply randomize IPs from above subnets, which is perfectly safe, that is always yields a valid private or public IP?

Even right now faker does not generate public IPs from all available public ranges, but because of the network math it employs it fails in the way that started this issue in the first place.
@maticomp Well, this is an internal python library. So with python 3, it's just there already. So it's best to use libraries that already exist, especially for cases like this. It would be best to fix those bugs in the internal libraries, than to just forget about it. 

In fact the ipaddress library already does what you are saying. There really is no such thing as public ip addresses,  it's that certain ranges were choosen to be reserved or non-routable. So having a hardcoded list of public ips doesn't make too much sense, but simply excluding the private or reserved ones would make more sense. 

I'm tempted to just extend the ipaddress class instead.
@kungfu71186 yes, but relying on public IPs being the inverse of private/reserved IPs makes it entirely dependent on the list of private/reserved subnets being accurate. I think we should submit a PR and hope they fix it, but I wouldn't be so sure. 

It's possible we might end up with having a list of subnets here in faker too for the time being.

On the other hand, having few public/private subnets that are not going nowhere anytime soon is virtually bullet-proof.

Either way, I do of course think that we should use `ipaddress` internally - at the very least because it gives us network math. Code can be greatly simplified if we rely on `ipaddress` to provide a subnet, and then need to only randomize IP matching that.
@maticomp I 100% agree in that we need to fix it on our end for right now. The issue right now is how that library is setup as well. 

Might have to do something like this:

```
from ipaddress import IPv4Network, IPv4Address, _IPv4Constants

IPv4Address._constants._private_networks += [IPv4Network('192.0.0.0/24')]

print(IPv4Address('192.0.0.145').is_private)
```

Edit, I do believe we can exclude private networks from all networks pretty easily. This is what I wanted to create a proof of concept for, but defining a network like `0.0.0.0/1` and then excluding the private/reserved networks should be possible. 

@kungfu71186 `address_exclude`  through `_constants._private_networks` works fine on `ip_network('0.0.0.0/0')` to get the full list of public subnets. It's a pity there `ipaddress` does not expose a neat way of randomizing an IP address inside a subnet. Most of those are huge, so getting a random item from `hosts()` generator would be slow, so we sill have to generate just like we do right now.

As for updating the faker constants, I'll post a PR for that.
@maticomp it's actually simpler than that. Don't need hosts. Just need to exclude the ranges.

privateNetwork = ip_network('0.0.0.0/0')

print(privateNetwork[0])
print(privateNetwork[4294767296])
print(privateNetwork[-1])
@kungfu71186 oh, right, I missed that it can be sliced like that. Let me work on the PR for that, I'll do some refactoring too to rely on `ipaddress` for whatever we can.
@maticomp Yeah, let me know what you come up with. I feel like the best way is to iterate over the constants to get ranges and almost use the same code that is there now 

`min_, max_ = self.generator.random.choice(net_ranges)` to get a random int. From there, you can simply get the ip address as discussed before.

To get a range, it's pretty easy with the ip_network class. Basically just:
`(int(ip_network('192.168.0.0/24')[0]), int(ip_network('192.168.0.0/24')[-1]))`

But in our case, we need to know all the public ips. It looks like in the current code, it just creates a list of ranges using the last ip address, all the way to the ip address right before the next network. 

Resolving patch:
diff --git a/faker/providers/internet/__init__.py b/faker/providers/internet/__init__.py
--- a/faker/providers/internet/__init__.py
+++ b/faker/providers/internet/__init__.py
@@ -5,7 +5,7 @@
 
 from .. import BaseProvider
 
-from ipaddress import ip_address, ip_network, IPV4LENGTH, IPV6LENGTH
+from ipaddress import ip_address, ip_network, IPv4Address, IPV4LENGTH, IPV6LENGTH
 
 # from faker.generator import random
 # from faker.providers.lorem.la import Provider as Lorem
@@ -15,36 +15,58 @@
 localized = True
 
 
-IPV4_PUBLIC_NETS = {
-    'a': ip_network('0.0.0.0/1'),
-    'b': ip_network('128.0.0.0/2'),
-    'c': ip_network('192.0.0.0/3')
-}
-IPV4_PRIVATE_NET_BLOCKS = {
-    'a': (
-        ip_network('0.0.0.0/8'),
+class _IPv4Constants:
+    """
+    IPv4 network constants used to group networks into different categories.
+    Structure derived from `ipaddress._IPv4Constants`.
+
+    Excluded network list is updated to comply with current IANA list of
+    private and reserved networks.
+    """
+    _network_classes = {
+        'a': ip_network('0.0.0.0/1'),
+        'b': ip_network('128.0.0.0/2'),
+        'c': ip_network('192.0.0.0/3')
+    }
+
+    _linklocal_network = ip_network('169.254.0.0/16')
+
+    _loopback_network = ip_network('127.0.0.0/8')
+
+    _multicast_network = ip_network('224.0.0.0/4')
+
+    # Three common private networks from class A, B and CIDR
+    # to generate private addresses from.
+    _private_networks = [
         ip_network('10.0.0.0/8'),
+        ip_network('172.16.0.0/12'),
+        ip_network('192.168.0.0/16')
+    ]
+
+    # List of networks from which IP addresses will never be generated,
+    # includes other private IANA and reserved networks from
+    # ttps://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml
+    _excluded_networks = [
+        ip_network('0.0.0.0/8'),
+        ip_network('100.64.0.0/10'),
         ip_network('127.0.0.0/8'),
-    ),
-    'b': (
         ip_network('169.254.0.0/16'),
-        ip_network('172.16.0.0/12')
-    ),
-    'c': (
-        ip_network('192.0.0.0/29'),
-        ip_network('192.0.0.170/31'),
+        ip_network('192.0.0.0/24'),
         ip_network('192.0.2.0/24'),
-        ip_network('192.168.0.0/16'),
+        ip_network('192.31.196.0/24'),
+        ip_network('192.52.193.0/24'),
+        ip_network('192.88.99.0/24'),
+        ip_network('192.175.48.0/24'),
         ip_network('198.18.0.0/15'),
         ip_network('198.51.100.0/24'),
-        ip_network('203.0.113.0/24')
-    ),
-}
-IPV4_NET_CLASSES = {
-    'a': (167772160, 184549375, 24),
-    'b': (2886729728, 2887778303, 20),
-    'c': (3232235520, 3232301055, 16)
-}
+        ip_network('203.0.113.0/24'),
+        ip_network('240.0.0.0/4'),
+        ip_network('255.255.255.255/32')
+    ] + [
+        _linklocal_network,
+        _loopback_network,
+        _multicast_network
+    ]
 
 
 class Provider(BaseProvider):
@@ -213,6 +235,80 @@ def url(self, schemes=None):
 
         return self.generator.parse(pattern)
 
+    def _random_ipv4_address_from_subnet(self, subnet, network=False):
+        """
+        Produces a random IPv4 address or network with a valid CIDR
+        from within a given subnet.
+
+        :param subnet: IPv4Network to choose from within
+        :param network: Return a network address, and not an IP address
+        """
+        address = str(
+            subnet[self.generator.random.randint(
+                0, subnet.num_addresses - 1
+            )]
+        )
+
+        if network:
+            address += '/' + str(self.generator.random.randint(
+                subnet.prefixlen,
+                subnet.max_prefixlen
+            ))
+            address = str(ip_network(address, strict=False))
+
+        return address
+
+    def _exclude_ipv4_networks(self, networks, networks_to_exclude):
+        """
+        Exclude the list of networks from another list of networks
+        and return a flat list of new networks.
+
+        :param networks: List of IPv4 networks to exclude from
+        :param networks_to_exclude: List of IPv4 networks to exclude
+        :returns: Flat list of IPv4 networks
+        """
+        for network_to_exclude in networks_to_exclude:
+            def _exclude_ipv4_network(network):
+                """
+                Exclude a single network from another single network
+                and return a list of networks. Network to exclude
+                comes from the outer scope.
+
+                :param network: Network to exclude from
+                :returns: Flat list of IPv4 networks after exclusion.
+                          If exclude fails because networks do not
+                          overlap, a single element list with the
+                          orignal network is returned. If it overlaps,
+                          even partially, the network is excluded.
+                """
+                try:
+                    return list(network.address_exclude(network_to_exclude))
+                except ValueError:
+                    # If networks overlap partially, `address_exclude`
+                    # will fail, but the network still must not be used
+                    # in generation.
+                    if network.overlaps(network_to_exclude):
+                        return []
+                    else:
+                        return [network]
+
+            networks = list(map(_exclude_ipv4_network, networks))
+
+            # flatten list of lists
+            networks = [
+                item for nested in networks for item in nested
+            ]
+
+        return networks
+
+    def ipv4_network_class(self):
+        """
+        Returns a IPv4 network class 'a', 'b' or 'c'.
+
+        :returns: IPv4 network class
+        """
+        return self.random_element('abc')
+
     def ipv4(self, network=False, address_class=None, private=None):
         """
         Produce a random IPv4 address or network with a valid CIDR.
@@ -228,24 +324,25 @@ def ipv4(self, network=False, address_class=None, private=None):
         elif private is False:
             return self.ipv4_public(address_class=address_class,
                                     network=network)
-        if address_class is None:
-            ip_range = (0, (2 ** IPV4LENGTH) - 1)
+
+        # if neither private nor public is required explicitly,
+        # generate from whole requested address space
+        if address_class:
+            all_networks = [_IPv4Constants._network_classes[address_class]]
         else:
-            net = IPV4_PUBLIC_NETS[address_class]
-            ip_range = (net._ip[0], net.ip[-1])
-        address = str(ip_address(self.generator.random.randint(*ip_range)))
-        if network:
-            address += '/' + str(self.generator.random.randint(0, IPV4LENGTH))
-            address = str(ip_network(address, strict=False))
-        return address
+            # if no address class is choosen, use whole IPv4 pool
+            all_networks = [ip_network('0.0.0.0/0')]
 
-    def ipv4_network_class(self):
-        """
-        Returns a IPv4 network class 'a', 'b' or 'c'.
+        # exclude special networks
+        all_networks = self._exclude_ipv4_networks(
+            all_networks,
+            _IPv4Constants._excluded_networks
+        )
 
-        :returns: IPv4 network class
-        """
-        return self.random_element('abc')
+        # choose random network from the list
+        random_network = self.generator.random.choice(all_networks)
+
+        return self._random_ipv4_address_from_subnet(random_network, network)
 
     def ipv4_private(self, network=False, address_class=None):
         """
@@ -255,14 +352,26 @@ def ipv4_private(self, network=False, address_class=None):
         :param address_class: IPv4 address class (a, b, or c)
         :returns: Private IPv4
         """
-        address_class = address_class or self.ipv4_network_class()
-        min_, max_, netmask = IPV4_NET_CLASSES[address_class]
-        address = str(ip_address(
-            self.generator.random.randint(min_, max_)))
-        if network:
-            address += '/' + str(self.generator.random.randint(netmask, 31))
-            address = str(ip_network(address, strict=False))
-        return address
+        # compute private networks from given class
+        supernet = _IPv4Constants._network_classes[
+            address_class or self.ipv4_network_class()
+        ]
+
+        private_networks = [
+            subnet for subnet in _IPv4Constants._private_networks
+            if subnet.overlaps(supernet)
+        ]
+
+        # exclude special networks
+        private_networks = self._exclude_ipv4_networks(
+            private_networks,
+            _IPv4Constants._excluded_networks
+        )
+
+        # choose random private network from the list
+        private_network = self.generator.random.choice(private_networks)
+
+        return self._random_ipv4_address_from_subnet(private_network, network)
 
     def ipv4_public(self, network=False, address_class=None):
         """
@@ -272,34 +381,22 @@ def ipv4_public(self, network=False, address_class=None):
         :param address_class: IPv4 address class (a, b, or c)
         :returns: Public IPv4
         """
-        address_class = address_class or self.ipv4_network_class()
-        public_net = IPV4_PUBLIC_NETS[address_class]
-        private_blocks = IPV4_PRIVATE_NET_BLOCKS[address_class]
-        # Make valid IP ranges, created from private block exclusion
-        net_ranges = []
-        ## Starts at end of 1st block if it's 1st of class
-        if public_net[0] != private_blocks[0][0]:
-            net_ranges = [(public_net[0]._ip, private_blocks[0][0]._ip-1)]
-        ## Loop on private blocks and guess available ranges
-        for i, block in enumerate(private_blocks):
-            if i+1 == len(private_blocks):
-                break
-            net_range = (block[-1]._ip+1, private_blocks[i+1][0]._ip-1)
-            net_ranges.append(net_range)
-        ## Add last range
-        net_ranges.append((private_blocks[-1][-1]._ip-1, public_net[-1]._ip-1))
-        net_ranges = [(i, j) for i, j in net_ranges if (j-i) > 0]
-        # Choose a range
-        min_, max_ = self.generator.random.choice(net_ranges)
-        address = str(ip_address(
-            self.generator.random.randint(min_, max_)))
-        # Add network mask
-        if network:
-            net_masks = {'a': 8, 'b': 16, 'c': 24}
-            min_net_mask = net_masks[address_class]
-            address += '/' + str(self.generator.random.randint(min_net_mask, 31))
-            address = str(ip_network(address, strict=False))
-        return address
+        # compute public networks
+        public_networks = [_IPv4Constants._network_classes[
+            address_class or self.ipv4_network_class()
+        ]]
+
+        # exclude private and excluded special networks
+        public_networks = self._exclude_ipv4_networks(
+            public_networks,
+            _IPv4Constants._private_networks +
+            _IPv4Constants._excluded_networks
+        )
+
+        # choose random public network from the list
+        public_network = self.generator.random.choice(public_networks)
+
+        return self._random_ipv4_address_from_subnet(public_network, network)
 
     def ipv6(self, network=False):
         """Produce a random IPv6 address or network with a valid CIDR"""


Test patch:
diff --git a/tests/test_factory.py b/tests/test_factory.py
--- a/tests/test_factory.py
+++ b/tests/test_factory.py
@@ -6,7 +6,7 @@
 import unittest
 import string
 import sys
-from ipaddress import ip_address, ip_network
+from ipaddress import ip_address, ip_network, IPv4Address
 
 import logging
 
@@ -441,7 +441,7 @@ def test_ipv4_network_class(self):
             self.assertIn(klass, 'abc')
 
     def test_ipv4_private(self):
-        from faker.providers.internet import Provider
+        from faker.providers.internet import Provider, _IPv4Constants
         provider = Provider(self.generator)
 
         for _ in range(999):
@@ -463,8 +463,12 @@ def test_ipv4_private(self):
                 re.compile(r'^(\d{1,3}\.){3}\d{1,3}/\d{1,2}$').search(address))
 
     def test_ipv4_private_class_a(self):
-        from faker.providers.internet import Provider
+        from faker.providers.internet import Provider, _IPv4Constants
         provider = Provider(self.generator)
+        
+        class_network = _IPv4Constants._network_classes['a']
+        class_min = class_network.network_address
+        class_max = class_network.broadcast_address
 
         for _ in range(999):
             address = provider.ipv4_private(address_class='a')
@@ -472,12 +476,16 @@ def test_ipv4_private_class_a(self):
             self.assertTrue(len(address) >= 7)
             self.assertTrue(len(address) <= 15)
             self.assertTrue(ip_address(address).is_private)
-            self.assertTrue(
-                re.compile(r'^10\.(\d{1,3}\.){2}\d{1,3}$').search(address))
+            self.assertTrue(ip_address(address) >= class_min)
+            self.assertTrue(ip_address(address) <= class_max)
 
     def test_ipv4_private_class_b(self):
-        from faker.providers.internet import Provider
+        from faker.providers.internet import Provider, _IPv4Constants
         provider = Provider(self.generator)
+        
+        class_network = _IPv4Constants._network_classes['b']
+        class_min = class_network.network_address
+        class_max = class_network.broadcast_address
 
         for _ in range(999):
             address = provider.ipv4_private(address_class='b')
@@ -485,12 +493,16 @@ def test_ipv4_private_class_b(self):
             self.assertTrue(len(address) >= 7)
             self.assertTrue(len(address) <= 15)
             self.assertTrue(ip_address(address).is_private)
-            self.assertTrue(
-                re.compile(r'^172\.(\d{1,3}\.){2}\d{1,3}$').search(address))
+            self.assertTrue(ip_address(address) >= class_min)
+            self.assertTrue(ip_address(address) <= class_max)
 
     def test_ipv4_private_class_c(self):
-        from faker.providers.internet import Provider
+        from faker.providers.internet import Provider, _IPv4Constants
         provider = Provider(self.generator)
+        
+        class_network = _IPv4Constants._network_classes['c']
+        class_min = class_network.network_address
+        class_max = class_network.broadcast_address
 
         for _ in range(999):
             address = provider.ipv4_private(address_class='c')
@@ -498,12 +510,12 @@ def test_ipv4_private_class_c(self):
             self.assertTrue(len(address) >= 7)
             self.assertTrue(len(address) <= 15)
             self.assertTrue(ip_address(address).is_private)
-            self.assertTrue(
-                re.compile(r'^192\.168\.\d{1,3}\.\d{1,3}$').search(address))
+            self.assertTrue(ip_address(address) >= class_min)
+            self.assertTrue(ip_address(address) <= class_max)
 
     def test_ipv4_public(self):
-        from faker.providers.internet import Provider
-        provider = Provider(self.generator)
+        from faker.providers.internet import Provider, _IPv4Constants
+        provider = Provider(self.generator)   
 
         for _ in range(999):
             address = provider.ipv4_public()