Repository: joke2k/faker

Issue:
Performance Optimizations
* Faker version: Master branch

* OS: Mac OS 

I am a Performance Engineer at Salesforce.org and I use Faker within my tool Snowfakery to generate hundreds of millions of rows of data. Some teams within my company have avoided Faker because they say it is too slow.

I'm a huge fan of Faker, but it is true that despite the complexity of Snowfakery overall (including writing to a SQL database), Faker is often the bottleneck. Luckily, it seems easy to fix.  I investigated and found some very quick wins in terms of performance.

If the person creating the Faker() object could have more influence on the random_element method, they could get a gigantic speedup. I understand that some people in some cases need the sophisticated distribution features of the underlying Faker library, but my users do not, so I wish I could turn them off.

### Steps to reproduce

```
from unittest.mock import patch
import random
from collections import OrderedDict

import faker
import timeit

def bench():
    f = faker.Faker()

    print(timeit.timeit(lambda:f.first_name(), number=100000))
        

def fast_random_element(self, choices):
    if isinstance(choices, OrderedDict):
        return random.choice(tuple(choices.keys()))
    else:
        return random.choice(choices)


def fast_random_element_2(self, choices):
    if isinstance(choices, OrderedDict):
        if not hasattr(choices, "_cached_choice_list"):
            setattr(choices, "_cached_choice_list", tuple(choices.keys()))
        choices = choices._cached_choice_list

    return random.choice(choices)

def all_bench():
    print("Warmup")
    bench()
    print("Normal")
    bench()
    with patch("faker.providers.BaseProvider.random_element", fast_random_element):
        from faker import providers
        print("Simple optimization - No caching")
        bench()

    with patch("faker.providers.BaseProvider.random_element", fast_random_element_2):
        print("With caching")
        bench()
    print("Baseline again")
    bench()

all_bench()
````

### Expected behaviour

Faker should be roughly as fast for simple element choices as Python itself.

### Actual behavior

These times are in seconds:

```
Faker
6.72312176
Simpler optimization
1.888246217999999
With caching
0.3221195019999996
```

I am willing to submit a PR for this but I might need some guidance about where you would want to store the "weighted or fast" flag. Perhaps pass it from the Faker() constructor to the Provider objects?

Discussion:
prescod: I think I found another optimization:

```
>>> from random import choice, choices
>>> timeit(lambda: choice([1,2,3,4]))
0.6210294770000075
>>> timeit(lambda: choice([1,2,3,4]))
0.6256943789999951
>>> timeit(lambda: choice([1,2,3,4]))
0.6176325039999995
>>> timeit(lambda: choices([1,2,3,4]))
1.126059282
>>> timeit(lambda: choices([1,2,3,4]))
1.1428683540000009
>>> timeit(lambda: choices([1,2,3,4]))
1.145066651999997
>>> timeit(lambda: choices([1,2,3,4]))
1.1160010509999978
```

prescod: Also, I'd suggest that going forward you look for opportunities to use tuples instead of lists. Tuples are more efficient and bug-resistent in a variety of ways.

```
>>> timeit(lambda: [1,2,3,4], number=10000000)
1.1680161589999898
>>> timeit(lambda: [1,2,3,4], number=10000000)
1.1769367479998891
>>> timeit(lambda: (1,2,3,4), number=10000000)
0.6921508979999089
>>> timeit(lambda: (1,2,3,4), number=10000000)
0.7061773149998771
```

```
>>> a = [1,2,3,4,5]
>>> b = list(a)
>>> a is b
False
>>> a = (1,2,3,4,5)
>>> b = tuple(a)
>>> a is b
True
```
prescod: I can't reproduce the test failures on my computer. :( Do you recognize them?

```
>       assert not errors, "Errors:\n - " + "\n - ".join(errors)
E       AssertionError: Errors:
E          - Attribute {{first_name_female}} provided an invalid value in format '{{prefix_female}} {{first_name_female}} {{last_name}}' from faker.providers.person.ar_PS.formats
E       assert not ["Attribute {{first_name_female}} provided an invalid value in format '{{prefix_female}} {{first_name_female}} {{last_name}}' from faker.providers.person.ar_PS.formats"]
```

fcurella: > I can't reproduce the test failures on my computer. :( Do you recognize them?
> 
> ```
> >       assert not errors, "Errors:\n - " + "\n - ".join(errors)
> E       AssertionError: Errors:
> E          - Attribute {{first_name_female}} provided an invalid value in format '{{prefix_female}} {{first_name_female}} {{last_name}}' from faker.providers.person.ar_PS.formats
> E       assert not ["Attribute {{first_name_female}} provided an invalid value in format '{{prefix_female}} {{first_name_female}} {{last_name}}' from faker.providers.person.ar_PS.formats"]
> ```

I don't have Windows, so I don't have a way to reproduce the failure.

At a first glance I don't see anything wrong with the code, nor the test. I'd like to have this issue resolved before we can merge, I don't want to break Faker for Windows user.

It seems to have been introduced in d867c31 ("Improve arg naming") and, from what I can tell, it randomly impact Windows 32 on different versions of Python (3.8, 3.9).


Patch:
diff --git a/.bumpversion.cfg b/.bumpversion.cfg
index ea106d8b..b2ec20af 100644
--- a/.bumpversion.cfg
+++ b/.bumpversion.cfg
@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 5.4.0
+current_version = 5.4.1
 files = VERSION faker/__init__.py docs/conf.py
 commit = True
 tag = True
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 66e76d5e..bcc57e37 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,18 +1,22 @@
 ## Changelog
 
-### [5.4.0 - 2021-01-07](https://github.com/joke2k/faker/compare/v5.4.0...v5.3.0)
+### [5.4.1 - 2021-01-11](https://github.com/joke2k/faker/compare/v5.4.0...v5.4.1)
+
+* Remove empty string from  ``ar_AA`` Person Provider.
+
+### [5.4.0 - 2021-01-07](https://github.com/joke2k/faker/compare/v5.3.0...v5.4.0)
 
 * Add ``da_DK`` address provider. Thanks @JoseNavy.
 
-### [5.3.0 - 2020-12-30](https://github.com/joke2k/faker/compare/v5.3.0...v5.2.0)
+### [5.3.0 - 2020-12-30](https://github.com/joke2k/faker/compare/v5.2.0...v5.3.0)
 
 * ``json`` and ``fixed_width`` now allow for strings to be fixed or pinned to a static value by prepending ``@``. Thanks @johnbrandborg.
 
-### [5.2.0 - 2020-12-30](https://github.com/joke2k/faker/compare/v5.2.0...v5.1.0)
+### [5.2.0 - 2020-12-30](https://github.com/joke2k/faker/compare/v5.1.0...v5.2.0)
 
 * Add ``en_IN`` address + phone number provider. Thanks @pulkitgupta2k.
 
-### [5.1.0 - 2020-12-28](https://github.com/joke2k/faker/compare/v5.1.0...v5.0.1)
+### [5.1.0 - 2020-12-28](https://github.com/joke2k/faker/compare/v5.0.1...v5.1.0)
 
 * Add ``en_IN`` person provider. Thanks @pulkitgupta2k.
 
diff --git a/README.rst b/README.rst
index 42aad2ba..ba40b1bb 100644
--- a/README.rst
+++ b/README.rst
@@ -165,6 +165,15 @@ providers package. The localization of Faker is an ongoing process, for
 which we need your help. Please don't hesitate to create a localized
 provider for your own locale and submit a Pull Request (PR).
 
+Optimizations
+-------------
+The Faker constructor takes a performance-related argument called
+``use_weighting``. It specifies whether to attempt to have the frequency
+of values match real-world frequencies (e.g. the English name Gary would
+be much more frequent than the name Lorimer). If ``use_weighting`` is ``False``,
+then all items have an equal chance of being selected, and the selection
+process is much faster. The default is ``True``.
+
 Command line usage
 ------------------
 
diff --git a/VERSION b/VERSION
index 8a30e8f9..ade65226 100644
--- a/VERSION
+++ b/VERSION
@@ -1,1 +1,1 @@
-5.4.0
+5.4.1
diff --git a/docs/conf.py b/docs/conf.py
index 6f589bfa..cf892092 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -51,9 +51,9 @@ copyright = '2014, Daniele Faraglia'
 # built documents.
 #
 # The short X.Y version.
-version = '5.4.0'
+version = '5.4.1'
 # The full version, including alpha/beta/rc tags.
-release = '5.4.0'
+release = '5.4.1'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
diff --git a/faker/__init__.py b/faker/__init__.py
index ef5651a2..62674b0d 100644
--- a/faker/__init__.py
+++ b/faker/__init__.py
@@ -2,6 +2,6 @@ from faker.factory import Factory
 from faker.generator import Generator
 from faker.proxy import Faker
 
-VERSION = '5.4.0'
+VERSION = '5.4.1'
 
 __all__ = ('Factory', 'Generator', 'Faker')
diff --git a/faker/factory.py b/faker/factory.py
index b898e3b2..b6a32fa1 100644
--- a/faker/factory.py
+++ b/faker/factory.py
@@ -27,6 +27,9 @@ class Factory:
             providers=None,
             generator=None,
             includes=None,
+            # Should we use weightings (more realistic) or weight every element equally (faster)?
+            # By default, use weightings for backwards compatibility & realism
+            use_weighting=True,
             **config):
         if includes is None:
             includes = []
@@ -39,6 +42,7 @@ class Factory:
             raise AttributeError(msg)
 
         config['locale'] = locale
+        config['use_weighting'] = use_weighting
         providers = providers or PROVIDERS
 
         providers += includes
@@ -51,6 +55,7 @@ class Factory:
 
             prov_cls, lang_found = cls._get_provider_class(prov_name, locale)
             provider = prov_cls(faker)
+            provider.__use_weighting__ = use_weighting
             provider.__provider__ = prov_name
             provider.__lang__ = lang_found
             faker.add_provider(provider)
diff --git a/faker/providers/__init__.py b/faker/providers/__init__.py
index ba1f0621..e976f157 100644
--- a/faker/providers/__init__.py
+++ b/faker/providers/__init__.py
@@ -17,6 +17,7 @@ class BaseProvider:
 
     __provider__ = 'base'
     __lang__ = None
+    __use_weighting__ = False
 
     # Locales supported by Linux Mint from `/usr/share/i18n/SUPPORTED`
     language_locale_codes = {
@@ -215,7 +216,8 @@ class BaseProvider:
         """
         return self.generator.random.choice(string.ascii_uppercase)
 
-    def random_elements(self, elements=('a', 'b', 'c'), length=None, unique=False):
+    def random_elements(self, elements=('a', 'b', 'c'), length=None, unique=False,
+                        use_weighting=None):
         """Generate a list of randomly sampled objects from ``elements``.
 
         Set ``unique`` to ``False`` for random sampling with replacement, and set ``unique`` to
@@ -271,6 +273,10 @@ class BaseProvider:
                        ("d", 0.05),
                    ]), unique=True
         """
+        use_weighting = (use_weighting
+                         if use_weighting is not None
+                         else self.__use_weighting__)
+
         if isinstance(elements, dict) and not isinstance(elements, OrderedDict):
             raise ValueError("Use OrderedDict only to avoid dependency on PYTHONHASHSEED (See #363).")
 
@@ -284,18 +290,21 @@ class BaseProvider:
                 "Sample length cannot be longer than the number of unique elements to pick from.")
 
         if isinstance(elements, dict):
-            choices = elements.keys()
-            probabilities = elements.values()
+            if not hasattr(elements, "_key_cache"):
+                elements._key_cache = tuple(elements.keys())
+
+            choices = elements._key_cache
+            probabilities = tuple(elements.values()) if use_weighting else None
         else:
             if unique:
                 # shortcut
                 return self.generator.random.sample(elements, length)
             choices = elements
-            probabilities = [1.0 for _ in range(len(choices))]
+            probabilities = None
 
         return fn(
-            list(choices),
-            list(probabilities),
+            tuple(choices),
+            probabilities,
             self.generator.random,
             length=length,
         )
diff --git a/faker/providers/person/ar_AA/__init__.py b/faker/providers/person/ar_AA/__init__.py
index 5951a23b..0b2cfd29 100644
--- a/faker/providers/person/ar_AA/__init__.py
+++ b/faker/providers/person/ar_AA/__init__.py
@@ -35,7 +35,7 @@ class Provider(PersonProvider):
         'بلقيس', 'بلماء', 'بلند', 'بنان', 'بنفسج', 'بهابهاء', 'بهية',
         'نوره', 'نوف', 'نوال', 'ناديه', 'هدى', 'هناء', 'هيا', 'هند',
         'هنادي', 'وفاء', 'ياسمين', 'يسرى', 'غيداء', 'شادن', 'جود',
-        'سلاف', 'جيلان', 'نشوة', 'ريان', 'دارين', '', 'أحلام', 'إخلاص',
+        'سلاف', 'جيلان', 'نشوة', 'ريان', 'دارين', 'أحلام', 'إخلاص',
         'أروى', 'أريج', 'أزهار', 'أسرار', 'آيات', 'ماذى', 'تولين',
         'هيام', 'ريناد', 'جميلة', 'حلا', 'عتاب', 'كرمة', 'ناهد', 'غوى',
         'ريف', 'بارعة', 'باسمة', 'باهرة', 'بتول', 'بثينة', 'أحمد',
diff --git a/faker/proxy.py b/faker/proxy.py
index d3910562..4303a073 100644
--- a/faker/proxy.py
+++ b/faker/proxy.py
@@ -24,7 +24,8 @@ class Faker:
     ]
 
     def __init__(self, locale=None, providers=None,
-                 generator=None, includes=None, **config):
+                 generator=None, includes=None,
+                 use_weighting=True, **config):
         self._factory_map = OrderedDict()
         self._weights = None
         self._unique_proxy = UniqueProxy(self)
@@ -56,7 +57,9 @@ class Faker:
             locales = [DEFAULT_LOCALE]
 
         for locale in locales:
-            self._factory_map[locale] = Factory.create(locale, providers, generator, includes, **config)
+            self._factory_map[locale] = Factory.create(locale, providers, generator, includes,
+                                                       use_weighting=use_weighting,
+                                                       **config)
 
         self._locales = locales
         self._factories = list(self._factory_map.values())
diff --git a/faker/utils/distribution.py b/faker/utils/distribution.py
index 02264e17..5d04cc26 100644
--- a/faker/utils/distribution.py
+++ b/faker/utils/distribution.py
@@ -1,7 +1,8 @@
 import bisect
+import itertools
 
 from random import Random
-from typing import Generator, Iterable, List, Optional, TypeVar
+from typing import Generator, Iterable, Optional, Sequence, TypeVar
 
 from faker.generator import random as mod_random
 
@@ -23,8 +24,8 @@ T = TypeVar('T')
 
 
 def choices_distribution_unique(
-        a: List[T], p: List[float], random: Optional[Random] = None, length: int = 1,
-) -> List[T]:
+        a: Sequence[T], p: Sequence[float], random: Optional[Random] = None, length: int = 1,
+) -> Sequence[T]:
     # As of Python 3.7, there isn't a way to sample unique elements that takes
     # weight into account.
     if random is None:
@@ -37,7 +38,7 @@ def choices_distribution_unique(
     items = list(a)
     probabilities = list(p)
     for i in range(length):
-        cdf = list(cumsum(probabilities))
+        cdf = tuple(cumsum(probabilities))
         normal = cdf[-1]
         cdf2 = [float(i) / float(normal) for i in cdf]
         uniform_sample = random_sample(random=random)
@@ -49,18 +50,26 @@ def choices_distribution_unique(
     return choices
 
 
-def choices_distribution(a: List[T], p: List[float], random: Optional[Random] = None, length: int = 1) -> List[T]:
+def choices_distribution(
+    a: Sequence[T], p: Sequence[float], random: Optional[Random] = None, length: int = 1,
+) -> Sequence[T]:
     if random is None:
         random = mod_random
 
-    assert len(a) == len(p)
+    if p is not None:
+        assert len(a) == len(p)
 
     if hasattr(random, 'choices'):
-        choices = random.choices(a, weights=p, k=length)
-        return choices
+        if length == 1 and p is None:
+            return (random.choice(a),)
+        else:
+            return random.choices(a, weights=p, k=length)
     else:
         choices = []
 
+        if p is None:
+            p = itertools.repeat(1, len(a))
+
         cdf = list(cumsum(p))
         normal = cdf[-1]
         cdf2 = [float(i) / float(normal) for i in cdf]


Test patch:
diff --git a/tests/providers/test_internet.py b/tests/providers/test_internet.py
index 9c025eae..468a595c 100644
--- a/tests/providers/test_internet.py
+++ b/tests/providers/test_internet.py
@@ -236,7 +236,7 @@ class TestInternetProvider:
         list_of_invalid_weights = [
             [1, 2, 3],   # List size does not match subnet list size
             ['a', 'b'],  # List size matches, but elements are invalid
-            None,        # Not a list or valid iterable
+            11,        # Not a list or valid iterable
         ]
 
         with patch('faker.providers.internet.choices_distribution',
diff --git a/tests/test_proxy.py b/tests/test_proxy.py
index 44eb73d5..38de6650 100644
--- a/tests/test_proxy.py
+++ b/tests/test_proxy.py
@@ -1,3 +1,5 @@
+import random
+
 from collections import OrderedDict
 from unittest.mock import PropertyMock, patch
 
@@ -318,6 +320,62 @@ class TestFakerProxyClass:
         with pytest.raises(AttributeError):
             fake.obviously_invalid_provider_method_a23f()
 
+    @patch('random.Random.choice')
+    @patch('random.Random.choices')
+    def test_weighting_disabled_single_choice(self, mock_choices_fn, mock_choice_fn):
+        fake = Faker(use_weighting=False)
+        fake.first_name()
+        mock_choice_fn.assert_called()
+        mock_choices_fn.assert_not_called()
+
+    @patch('random.Random.choice')
+    @patch('random.Random.choices', wraps=random.Random().choices)
+    def test_weighting_disabled_with_locales(self, mock_choices_fn, mock_choice_fn):
+        locale = OrderedDict([
+            ('de_DE', 3),
+            ('en-US', 2),
+            ('en-PH', 1),
+            ('ja_JP', 5),
+        ])
+        fake = Faker(locale, use_weighting=False)
+        fake.first_name()
+        mock_choices_fn.assert_called()  # select provider
+        mock_choice_fn.assert_called()   # select within provider
+
+    @patch('random.Random.choice')
+    @patch('random.Random.choices', wraps=random.Random().choices)
+    def test_weighting_disabled_multiple_locales(self, mock_choices_fn, mock_choice_fn):
+        locale = OrderedDict([
+            ('de_DE', 3),
+            ('en-US', 2),
+            ('en-PH', 1),
+            ('ja_JP', 5),
+        ])
+        fake = Faker(locale, use_weighting=False)
+        fake.first_name()
+        mock_choices_fn.assert_called()  # select provider
+        mock_choice_fn.assert_called()   # select within provider
+
+    @patch('random.Random.choice')
+    @patch('random.Random.choices', wraps=random.Random().choices)
+    def test_weighting_disabled_multiple_choices(self, mock_choices_fn, mock_choice_fn):
+        fake = Faker(use_weighting=False)
+        fake.uri_path(deep=3)
+
+        assert mock_choices_fn.mock_calls[0][2]["k"] == 3
+        assert mock_choices_fn.mock_calls[0][2]["weights"] is None
+        mock_choice_fn.assert_not_called()
+
+    @patch('random.Random.choice')
+    @patch('random.Random.choices', wraps=random.Random().choices)
+    def test_weighting_enabled_multiple_choices(self, mock_choices_fn, mock_choice_fn):
+        fake = Faker(use_weighting=True)
+        fake.uri_path(deep=3)
+
+        assert mock_choices_fn.mock_calls[0][2]["k"] == 3
+        assert mock_choices_fn.mock_calls[0][2]["weights"] is None
+        mock_choice_fn.assert_not_called()
+
     def test_dir_include_all_providers_attribute_in_list(self):
         fake = Faker(['en_US', 'en_PH'])
         expected = set(dir(Faker) + [