Repository: joke2k/faker

Issue:
ipv4 faker very slow
Hi! We updated from Faker 0.9.0 to 0.9.2 and noticed our tests speed decreased by 5X. We tracked the bug to the changes in the ipv4 faker done in commit 3b847f48bbfc33aed3e86cffcedbcfd1c0cccec5

### Steps to reproduce

1. checkout to commit 3b847f48bbfc33aed3e86cffcedbcfd1c0cccec5
1. run the following: `time python -c 'import faker; fake = faker.Faker(); [fake.ipv4() for _ in range(100)]'`
1. Check its output to see it was slow. in my laptop it took about 6 seconds.
1. Run `git checkout HEAD~1` to go to a working commit
1. Run the command of step 2 again and it will work much faster (in my case it took 0.4 seconds)

### Expected behavior

Generating ivp4 addresses should be fast

### Actual behavior

Generating ipv4 addresses is slow



Discussion:
We could add a flag to give users the choice between accuracy and speed, but I'd default it to be slow and correct.
I think the performance hit here would when it's building the network. Because you are doing this 100 times, it could definitely impact it. I don't think I'll have time to take a look right now, but if you define a class or range instead of the whole spectrum, it should speed it up.

Unfortunately I won't even be able to take a look for a few weeks if anything
I did a quick profiling.  It showed that calls to `_exclude_ipv4_network` are expensive.  Hope it would be helpful in later optimization.

```python
import cProfile
import pstats
cProfile.run('import faker; fake = faker.Faker(); [fake.ipv4() for _ in range(100)]', 'faker_stats')
pstats.Stats('faker_stats').strip_dirs().sort_stats('tottime').print_stats(20)
pstats.Stats('faker_stats').strip_dirs().sort_stats(pstats.SortKey.CUMULATIVE).print_stats(20)
```

the output is
```
Sun Jan 20 10:06:38 2019    faker_stats

         8320383 function calls (8319474 primitive calls) in 3.815 seconds

   Ordered by: internal time
   List reduced from 685 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   574400    0.623    0.000    0.896    0.000 ipaddress.py:670(__contains__)
   290062    0.298    0.000    0.298    0.000 {method 'join' of 'str' objects}
   858200    0.267    0.000    0.442    0.000 ipaddress.py:690(broadcast_address)
   160500    0.266    0.000    2.220    0.000 ipaddress.py:736(address_exclude)
   287200    0.219    0.000    0.931    0.000 ipaddress.py:613(__str__)
   143600    0.195    0.000    1.203    0.000 ipaddress.py:683(overlaps)
   288700    0.183    0.000    0.544    0.000 ipaddress.py:1179(_string_from_ip_int)
   145100    0.180    0.000    3.603    0.000 __init__.py:289(_exclude_ipv4_network)
   164200    0.178    0.000    0.676    0.000 ipaddress.py:975(_is_subnet_of)
  1180698    0.145    0.000    0.145    0.000 {built-in method builtins.isinstance}
   298300    0.137    0.000    0.167    0.000 ipaddress.py:560(__lt__)
   288700    0.132    0.000    0.676    0.000 ipaddress.py:585(__str__)
  1078896    0.114    0.000    0.114    0.000 ipaddress.py:550(__int__)
    32343    0.092    0.000    0.172    0.000 ipaddress.py:1477(__init__)
   904727    0.085    0.000    0.085    0.000 {method 'get' of 'dict' objects}
   288700    0.064    0.000    0.064    0.000 {method 'to_bytes' of 'int' objects}
   164200    0.060    0.000    0.160    0.000 functools.py:98(_le_from_lt)
   164200    0.056    0.000    0.732    0.000 ipaddress.py:987(subnet_of)
    46200    0.056    0.000    0.214    0.000 ipaddress.py:869(subnets)
    93918    0.046    0.000    0.071    0.000 ipaddress.py:1268(__init__)


Sun Jan 20 10:06:38 2019    faker_stats

         8320383 function calls (8319474 primitive calls) in 3.815 seconds

   Ordered by: cumulative time
   List reduced from 685 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    101/1    0.001    0.000    3.816    3.816 {built-in method builtins.exec}
        1    0.002    0.002    3.663    3.663 <string>:1(<listcomp>)
      100    0.001    0.000    3.660    0.037 __init__.py:330(ipv4)
      100    0.039    0.000    3.655    0.037 __init__.py:279(_exclude_ipv4_networks)
   145100    0.180    0.000    3.603    0.000 __init__.py:289(_exclude_ipv4_network)
   160500    0.266    0.000    2.220    0.000 ipaddress.py:736(address_exclude)
   143600    0.195    0.000    1.203    0.000 ipaddress.py:683(overlaps)
   287200    0.219    0.000    0.931    0.000 ipaddress.py:613(__str__)
   574400    0.623    0.000    0.896    0.000 ipaddress.py:670(__contains__)
   164200    0.056    0.000    0.732    0.000 ipaddress.py:987(subnet_of)
   164200    0.178    0.000    0.676    0.000 ipaddress.py:975(_is_subnet_of)
   288700    0.132    0.000    0.676    0.000 ipaddress.py:585(__str__)
   288700    0.183    0.000    0.544    0.000 ipaddress.py:1179(_string_from_ip_int)
   858200    0.267    0.000    0.442    0.000 ipaddress.py:690(broadcast_address)
   290062    0.298    0.000    0.298    0.000 {method 'join' of 'str' objects}
    46200    0.056    0.000    0.214    0.000 ipaddress.py:869(subnets)
    32343    0.092    0.000    0.172    0.000 ipaddress.py:1477(__init__)
   298300    0.137    0.000    0.167    0.000 ipaddress.py:560(__lt__)
   164200    0.060    0.000    0.160    0.000 functools.py:98(_le_from_lt)
  1180698    0.145    0.000    0.145    0.000 {built-in method builtins.isinstance}
```


Resolving patch:
diff --git a/faker/providers/internet/__init__.py b/faker/providers/internet/__init__.py
--- a/faker/providers/internet/__init__.py
+++ b/faker/providers/internet/__init__.py
@@ -10,6 +10,7 @@
 # from faker.generator import random
 # from faker.providers.lorem.la import Provider as Lorem
 from faker.utils.decorators import lowercase, slugify, slugify_unicode
+from faker.utils.distribution import choices_distribution
 
 
 localized = True
@@ -29,12 +30,6 @@ class _IPv4Constants:
         'c': ip_network('192.0.0.0/3'),
     }
 
-    _linklocal_network = ip_network('169.254.0.0/16')
-
-    _loopback_network = ip_network('127.0.0.0/8')
-
-    _multicast_network = ip_network('224.0.0.0/4')
-
     # Three common private networks from class A, B and CIDR
     # to generate private addresses from.
     _private_networks = [
@@ -49,8 +44,8 @@ class _IPv4Constants:
     _excluded_networks = [
         ip_network('0.0.0.0/8'),
         ip_network('100.64.0.0/10'),
-        ip_network('127.0.0.0/8'),
-        ip_network('169.254.0.0/16'),
+        ip_network('127.0.0.0/8'),  # loopback network
+        ip_network('169.254.0.0/16'),  # linklocal network
         ip_network('192.0.0.0/24'),
         ip_network('192.0.2.0/24'),
         ip_network('192.31.196.0/24'),
@@ -60,12 +55,9 @@ class _IPv4Constants:
         ip_network('198.18.0.0/15'),
         ip_network('198.51.100.0/24'),
         ip_network('203.0.113.0/24'),
+        ip_network('224.0.0.0/4'),  # multicast network
         ip_network('240.0.0.0/4'),
         ip_network('255.255.255.255/32'),
-    ] + [
-        _linklocal_network,
-        _loopback_network,
-        _multicast_network,
     ]
 
 
@@ -251,14 +243,123 @@ def url(self, schemes=None):
 
         return self.generator.parse(pattern)
 
-    def _random_ipv4_address_from_subnet(self, subnet, network=False):
+    def _get_all_networks_and_weights(self, address_class=None):
+        """
+        Produces a 2-tuple of valid IPv4 networks and corresponding relative weights
+
+        :param address_class: IPv4 address class (a, b, or c)
+        """
+        # If `address_class` has an unexpected value, use the whole IPv4 pool
+        if address_class in _IPv4Constants._network_classes.keys():
+            networks_attr = '_cached_all_class_{}_networks'.format(address_class)
+            all_networks = [_IPv4Constants._network_classes[address_class]]
+        else:
+            networks_attr = '_cached_all_networks'
+            all_networks = [ip_network('0.0.0.0/0')]
+
+        # Return cached network and weight data if available
+        weights_attr = '{}_weights'.format(networks_attr)
+        if hasattr(self, networks_attr) and hasattr(self, weights_attr):
+            return getattr(self, networks_attr), getattr(self, weights_attr)
+
+        # Otherwise, compute for list of networks (excluding special networks)
+        all_networks = self._exclude_ipv4_networks(
+            all_networks,
+            _IPv4Constants._excluded_networks,
+        )
+
+        # Then compute for list of corresponding relative weights
+        weights = [network.num_addresses for network in all_networks]
+
+        # Then cache and return results
+        setattr(self, networks_attr, all_networks)
+        setattr(self, weights_attr, weights)
+        return all_networks, weights
+
+    def _get_private_networks_and_weights(self, address_class=None):
+        """
+        Produces an OrderedDict of valid private IPv4 networks and corresponding relative weights
+
+        :param address_class: IPv4 address class (a, b, or c)
+        """
+        # If `address_class` has an unexpected value, choose a valid value at random
+        if address_class not in _IPv4Constants._network_classes.keys():
+            address_class = self.ipv4_network_class()
+
+        # Return cached network and weight data if available for a specific address class
+        networks_attr = '_cached_private_class_{}_networks'.format(address_class)
+        weights_attr = '{}_weights'.format(networks_attr)
+        if hasattr(self, networks_attr) and hasattr(self, weights_attr):
+            return getattr(self, networks_attr), getattr(self, weights_attr)
+
+        # Otherwise, compute for list of private networks (excluding special networks)
+        supernet = _IPv4Constants._network_classes[address_class]
+        private_networks = [
+            subnet for subnet in _IPv4Constants._private_networks
+            if subnet.overlaps(supernet)
+        ]
+        private_networks = self._exclude_ipv4_networks(
+            private_networks,
+            _IPv4Constants._excluded_networks,
+        )
+
+        # Then compute for list of corresponding relative weights
+        weights = [network.num_addresses for network in private_networks]
+
+        # Then cache and return results
+        setattr(self, networks_attr, private_networks)
+        setattr(self, weights_attr, weights)
+        return private_networks, weights
+
+    def _get_public_networks_and_weights(self, address_class=None):
+        """
+        Produces a 2-tuple of valid public IPv4 networks and corresponding relative weights
+
+        :param address_class: IPv4 address class (a, b, or c)
+        """
+        # If `address_class` has an unexpected value, choose a valid value at random
+        if address_class not in _IPv4Constants._network_classes.keys():
+            address_class = self.ipv4_network_class()
+
+        # Return cached network and weight data if available for a specific address class
+        networks_attr = '_cached_public_class_{}_networks'.format(address_class)
+        weights_attr = '{}_weights'.format(networks_attr)
+        if hasattr(self, networks_attr) and hasattr(self, weights_attr):
+            return getattr(self, networks_attr), getattr(self, weights_attr)
+
+        # Otherwise, compute for list of public networks (excluding private and special networks)
+        public_networks = [_IPv4Constants._network_classes[address_class]]
+        public_networks = self._exclude_ipv4_networks(
+            public_networks,
+            _IPv4Constants._private_networks +
+            _IPv4Constants._excluded_networks,
+        )
+
+        # Then compute for list of corresponding relative weights
+        weights = [network.num_addresses for network in public_networks]
+
+        # Then cache and return results
+        setattr(self, networks_attr, public_networks)
+        setattr(self, weights_attr, weights)
+        return public_networks, weights
+
+    def _random_ipv4_address_from_subnets(self, subnets, weights=None, network=False):
         """
         Produces a random IPv4 address or network with a valid CIDR
-        from within a given subnet.
+        from within the given subnets using a distribution described
+        by weights.
 
-        :param subnet: IPv4Network to choose from within
+        :param subnets: List of IPv4Networks to choose from within
+        :param weights: List of weights corresponding to the individual IPv4Networks
         :param network: Return a network address, and not an IP address
+        :return:
         """
+        # If the weights argument has an invalid value, default to equal distribution
+        try:
+            subnet = choices_distribution(subnets, weights, random=self.generator.random, length=1)[0]
+        except (AssertionError, TypeError):
+            subnet = self.generator.random.choice(subnets)
+
         address = str(
             subnet[self.generator.random.randint(
                 0, subnet.num_addresses - 1,
@@ -283,6 +384,7 @@ def _exclude_ipv4_networks(self, networks, networks_to_exclude):
         :param networks_to_exclude: List of IPv4 networks to exclude
         :returns: Flat list of IPv4 networks
         """
+        networks_to_exclude.sort(key=lambda x: x.prefixlen)
         for network_to_exclude in networks_to_exclude:
             def _exclude_ipv4_network(network):
                 """
@@ -327,7 +429,7 @@ def ipv4_network_class(self):
 
     def ipv4(self, network=False, address_class=None, private=None):
         """
-        Produce a random IPv4 address or network with a valid CIDR.
+        Returns a random IPv4 address or network with a valid CIDR.
 
         :param network: Network address
         :param address_class: IPv4 address class (a, b, or c)
@@ -340,25 +442,9 @@ def ipv4(self, network=False, address_class=None, private=None):
         elif private is False:
             return self.ipv4_public(address_class=address_class,
                                     network=network)
-
-        # if neither private nor public is required explicitly,
-        # generate from whole requested address space
-        if address_class:
-            all_networks = [_IPv4Constants._network_classes[address_class]]
         else:
-            # if no address class is choosen, use whole IPv4 pool
-            all_networks = [ip_network('0.0.0.0/0')]
-
-        # exclude special networks
-        all_networks = self._exclude_ipv4_networks(
-            all_networks,
-            _IPv4Constants._excluded_networks,
-        )
-
-        # choose random network from the list
-        random_network = self.generator.random.choice(all_networks)
-
-        return self._random_ipv4_address_from_subnet(random_network, network)
+            all_networks, weights = self._get_all_networks_and_weights(address_class=address_class)
+            return self._random_ipv4_address_from_subnets(all_networks, weights=weights, network=network)
 
     def ipv4_private(self, network=False, address_class=None):
         """
@@ -368,26 +454,8 @@ def ipv4_private(self, network=False, address_class=None):
         :param address_class: IPv4 address class (a, b, or c)
         :returns: Private IPv4
         """
-        # compute private networks from given class
-        supernet = _IPv4Constants._network_classes[
-            address_class or self.ipv4_network_class()
-        ]
-
-        private_networks = [
-            subnet for subnet in _IPv4Constants._private_networks
-            if subnet.overlaps(supernet)
-        ]
-
-        # exclude special networks
-        private_networks = self._exclude_ipv4_networks(
-            private_networks,
-            _IPv4Constants._excluded_networks,
-        )
-
-        # choose random private network from the list
-        private_network = self.generator.random.choice(private_networks)
-
-        return self._random_ipv4_address_from_subnet(private_network, network)
+        private_networks, weights = self._get_private_networks_and_weights(address_class=address_class)
+        return self._random_ipv4_address_from_subnets(private_networks, weights=weights, network=network)
 
     def ipv4_public(self, network=False, address_class=None):
         """
@@ -397,22 +465,8 @@ def ipv4_public(self, network=False, address_class=None):
         :param address_class: IPv4 address class (a, b, or c)
         :returns: Public IPv4
         """
-        # compute public networks
-        public_networks = [_IPv4Constants._network_classes[
-            address_class or self.ipv4_network_class()
-        ]]
-
-        # exclude private and excluded special networks
-        public_networks = self._exclude_ipv4_networks(
-            public_networks,
-            _IPv4Constants._private_networks +
-            _IPv4Constants._excluded_networks,
-        )
-
-        # choose random public network from the list
-        public_network = self.generator.random.choice(public_networks)
-
-        return self._random_ipv4_address_from_subnet(public_network, network)
+        public_networks, weights = self._get_public_networks_and_weights(address_class=address_class)
+        return self._random_ipv4_address_from_subnets(public_networks, weights=weights, network=network)
 
     def ipv6(self, network=False):
         """Produce a random IPv6 address or network with a valid CIDR"""


Test patch:
diff --git a/tests/test_factory.py b/tests/test_factory.py
--- a/tests/test_factory.py
+++ b/tests/test_factory.py
@@ -6,6 +6,10 @@
 import string
 import sys
 import unittest
+try:
+    from unittest.mock import patch, PropertyMock
+except ImportError:
+    from mock import patch, PropertyMock
 
 from collections import OrderedDict
 from ipaddress import ip_address, ip_network
@@ -526,6 +530,41 @@ def test_email(self):
             email = factory.email()
             assert '@' in email
 
+    def test_ipv4_caching(self):
+        from faker.providers.internet import Provider, _IPv4Constants
+
+        # The extra [None] here is to test code path involving whole IPv4 pool
+        for address_class in list(_IPv4Constants._network_classes.keys()) + [None]:
+            if address_class is None:
+                networks_attr = '_cached_all_networks'
+            else:
+                networks_attr = '_cached_all_class_{}_networks'.format(address_class)
+            weights_attr = '{}_weights'.format(networks_attr)
+            provider = Provider(self.generator)
+
+            # First, test cache creation
+            assert not hasattr(provider, networks_attr)
+            assert not hasattr(provider, weights_attr)
+            provider.ipv4(address_class=address_class)
+            assert hasattr(provider, networks_attr)
+            assert hasattr(provider, weights_attr)
+
+            # Then, test cache access on subsequent calls
+            with patch.object(Provider, networks_attr, create=True,
+                              new_callable=PropertyMock) as mock_networks_cache:
+                with patch.object(Provider, weights_attr, create=True,
+                                  new_callable=PropertyMock) as mock_weights_cache:
+                    # Keep test fast by patching the cache attributes to return something simple
+                    mock_networks_cache.return_value = [ip_network('10.0.0.0/24')]
+                    mock_weights_cache.return_value = [10]
+                    for _ in range(100):
+                        provider.ipv4(address_class=address_class)
+
+                    # Python's hasattr() internally calls getattr()
+                    # So each call to ipv4() accesses the cache attributes twice
+                    assert mock_networks_cache.call_count == 200
+                    assert mock_weights_cache.call_count == 200
+
     def test_ipv4(self):
         from faker.providers.internet import Provider
 
@@ -565,6 +604,37 @@ def test_ipv4_network_class(self):
             klass = provider.ipv4_network_class()
             assert klass in 'abc'
 
+    def test_ipv4_private_caching(self):
+        from faker.providers.internet import Provider, _IPv4Constants
+
+        for address_class in _IPv4Constants._network_classes.keys():
+            networks_attr = '_cached_private_class_{}_networks'.format(address_class)
+            weights_attr = '{}_weights'.format(networks_attr)
+            provider = Provider(self.generator)
+
+            # First, test cache creation
+            assert not hasattr(provider, networks_attr)
+            assert not hasattr(provider, weights_attr)
+            provider.ipv4_private(address_class=address_class)
+            assert hasattr(provider, networks_attr)
+            assert hasattr(provider, weights_attr)
+
+            # Then, test cache access on subsequent calls
+            with patch.object(Provider, networks_attr, create=True,
+                              new_callable=PropertyMock) as mock_networks_cache:
+                with patch.object(Provider, weights_attr, create=True,
+                                  new_callable=PropertyMock) as mock_weights_cache:
+                    # Keep test fast by patching the cache attributes to return something simple
+                    mock_networks_cache.return_value = [ip_network('10.0.0.0/24')]
+                    mock_weights_cache.return_value = [10]
+                    for _ in range(100):
+                        provider.ipv4_private(address_class=address_class)
+
+                    # Python's hasattr() internally calls getattr()
+                    # So each call to ipv4_private() accesses the cache attributes twice
+                    assert mock_networks_cache.call_count == 200
+                    assert mock_weights_cache.call_count == 200
+
     def test_ipv4_private(self):
         from faker.providers.internet import Provider
         provider = Provider(self.generator)
@@ -638,6 +708,37 @@ def test_ipv4_private_class_c(self):
             assert ip_address(address) >= class_min
             assert ip_address(address) <= class_max
 
+    def test_ipv4_public_caching(self):
+        from faker.providers.internet import Provider, _IPv4Constants
+
+        for address_class in _IPv4Constants._network_classes.keys():
+            networks_attr = '_cached_public_class_{}_networks'.format(address_class)
+            weights_attr = '{}_weights'.format(networks_attr)
+            provider = Provider(self.generator)
+
+            # First, test cache creation
+            assert not hasattr(provider, networks_attr)
+            assert not hasattr(provider, weights_attr)
+            provider.ipv4_public(address_class=address_class)
+            assert hasattr(provider, networks_attr)
+            assert hasattr(provider, weights_attr)
+
+            # Then, test cache access on subsequent calls
+            with patch.object(Provider, networks_attr, create=True,
+                              new_callable=PropertyMock) as mock_networks_cache:
+                with patch.object(Provider, weights_attr, create=True,
+                                  new_callable=PropertyMock) as mock_weights_cache:
+                    # Keep test fast by patching the cache attributes to return something simple
+                    mock_networks_cache.return_value = [ip_network('10.0.0.0/24')]
+                    mock_weights_cache.return_value = [10]
+                    for _ in range(100):
+                        provider.ipv4_public(address_class=address_class)
+
+                    # Python's hasattr() internally calls getattr()
+                    # So each call to ipv4_public() accesses the cache attributes twice
+                    assert mock_networks_cache.call_count == 200
+                    assert mock_weights_cache.call_count == 200
+
     def test_ipv4_public(self):
         from faker.providers.internet import Provider
         provider = Provider(self.generator)
@@ -697,6 +798,39 @@ def test_ipv4_public_class_c(self):
             assert len(address) <= 15
             assert not ip_address(address).is_private, address
 
+    def test_ipv4_distribution_selection(self):
+        from faker.providers.internet import Provider
+        from faker.utils.distribution import choices_distribution
+        provider = Provider(self.generator)
+
+        subnets = [ip_network('10.0.0.0/8'), ip_network('11.0.0.0/8')]
+        valid_weights = [1, 1]
+        list_of_invalid_weights = [
+            [1, 2, 3],   # List size does not match subnet list size
+            ['a', 'b'],  # List size matches, but elements are invalid
+            None,        # Not a list or valid iterable
+        ]
+
+        with patch('faker.providers.internet.choices_distribution',
+                   wraps=choices_distribution) as mock_choices_fn:
+            with patch('faker.generator.random.choice',
+                       wraps=random.choice) as mock_random_choice:
+                # If weights argument is valid, only `choices_distribution` should be called
+                provider._random_ipv4_address_from_subnets(subnets, valid_weights)
+                assert mock_choices_fn.call_count == 1
+                assert mock_random_choice.call_count == 0
+
+                # If weights argument is invalid, calls to `choices_distribution` will fail
+                # and calls to `random.choice` will be made as failover behavior
+                for invalid_weights in list_of_invalid_weights:
+                    # Reset mock objects for each iteration
+                    mock_random_choice.reset_mock()
+                    mock_choices_fn.reset_mock()
+
+                    provider._random_ipv4_address_from_subnets(subnets, invalid_weights)
+                    assert mock_choices_fn.call_count == 1
+                    assert mock_random_choice.call_count == 1
+
     def test_ipv6(self):
         from faker.providers.internet import Provider