Repository: joke2k/faker

Issue:
RecursionError with spark udf
* Faker version: 8.10.0
* OS: Ubuntu 18.04.5 LTS

I have covered the [issue in presidio library](https://github.com/microsoft/presidio/issues/741) but some time later I realized that the problem may lay here. I am trying to use the Faker inside the function which is converted to pyspark udf. After replacing the Faker part of the function everything works. And according to the stacktrace, the problem is related to the usage of the Faker.

### Steps to reproduce

1. Initialize fake `fake = Faker(locale=['en_US', 'en_GB', 'en_CA', 'fr_FR'])`
2. Initialize fake operators used inside microsoft presidio `fake_operators = {
    "PERSON": OperatorConfig("custom", {"lambda": lambda x: anon_suffix(fake.name)()}),
    "MY_NAME_IS": OperatorConfig("custom", {"lambda": lambda x: "name is " + anon_suffix(fake.name)()}),
    "PHONE_NUMBER": OperatorConfig("custom", {"lambda": lambda x: anon_suffix(fake.phone_number)()}),
    "EMAIL_ADDRESS": OperatorConfig("custom", {"lambda": lambda x: anon_suffix(fake.email)()}),
    "LOCATION": OperatorConfig("replace", {"new_value": "USA"}),
    "CAP_NUM_SEQ": OperatorConfig("custom", {"lambda": lambda x: anon_suffix(cap_num_generate)(x)}),
    "URL": OperatorConfig("custom", {"lambda": lambda x: anon_suffix(fake.url)()})
}`
3. Convert pandas apply to udf `anonymize_udf = pandas_udf(anonymize_series, returnType=StringType())`
4. Apply the udf `anonymized_df = input_df.withColumn('text_anonymized', anonymize_udf(F.col('text')))`
5. Convert the Spark DataFrame back to a pandas DataFrame using Arrow `df = anonymized_df.select("*").toPandas()`

### Expected behavior

It was expected that the udf works in the same manner and the pandas version. And it does when the Faker is not used.

### Actual behavior

Similarly to #1452
```
/databricks/spark/python/pyspark/sql/pandas/conversion.py:145: UserWarning: toPandas attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true, but has reached the error below and can not continue. Note that 'spark.sql.execution.arrow.pyspark.fallback.enabled' does not have an effect on failures in the middle of computation.
  An error occurred while calling o1299.getResult.
: org.apache.spark.SparkException: Exception thrown in awaitResult: 
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:428)
	at org.apache.spark.security.SocketAuthServer.getResult(SocketAuthServer.scala:107)
	at org.apache.spark.security.SocketAuthServer.getResult(SocketAuthServer.scala:103)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
	at py4j.Gateway.invoke(Gateway.java:295)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:251)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 0.0 failed 1 times, most recent failure: Lost task 3.0 in stage 0.0 (TID 3) (ip-10-145-235-196.ec2.internal executor driver): org.apache.spark.api.python.PythonException: 'pyspark.serializers.SerializationError: Caused by Traceback (most recent call last):
  File "/databricks/spark/python/pyspark/serializers.py", line 165, in _read_with_length
    return self.loads(obj)
  File "/databricks/spark/python/pyspark/serializers.py", line 469, in loads
    return pickle.loads(obj, encoding=encoding)
  File "/databricks/python/lib/python3.8/site-packages/faker/proxy.py", line 277, in __getattr__
    obj = getattr(self._proxy, name)
  File "/databricks/python/lib/python3.8/site-packages/faker/proxy.py", line 277, in __getattr__
    obj = getattr(self._proxy, name)
  File "/databricks/python/lib/python3.8/site-packages/faker/proxy.py", line 277, in __getattr__
    obj = getattr(self._proxy, name)
  [Previous line repeated 984 more times]
RecursionError: maximum recursion depth exceeded'. Full traceback below:
Traceback (most recent call last):
  File "/databricks/spark/python/pyspark/serializers.py", line 165, in _read_with_length
    return self.loads(obj)
  File "/databricks/spark/python/pyspark/serializers.py", line 469, in loads
    return pickle.loads(obj, encoding=encoding)
  File "/databricks/python/lib/python3.8/site-packages/faker/proxy.py", line 277, in __getattr__
    obj = getattr(self._proxy, name)
  File "/databricks/python/lib/python3.8/site-packages/faker/proxy.py", line 277, in __getattr__
    obj = getattr(self._proxy, name)
  File "/databricks/python/lib/python3.8/site-packages/faker/proxy.py", line 277, in __getattr__
    obj = getattr(self._proxy, name)
  [Previous line repeated 984 more times]
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

pyspark.serializers.SerializationError: Caused by Traceback (most recent call last):
  File "/databricks/spark/python/pyspark/serializers.py", line 165, in _read_with_length
    return self.loads(obj)
  File "/databricks/spark/python/pyspark/serializers.py", line 469, in loads
    return pickle.loads(obj, encoding=encoding)
  File "/databricks/python/lib/python3.8/site-packages/faker/proxy.py", line 277, in __getattr__
    obj = getattr(self._proxy, name)
  File "/databricks/python/lib/python3.8/site-packages/faker/proxy.py", line 277, in __getattr__
    obj = getattr(self._proxy, name)
  File "/databricks/python/lib/python3.8/site-packages/faker/proxy.py", line 277, in __getattr__
    obj = getattr(self._proxy, name)
  [Previous line repeated 984 more times]
RecursionError: maximum recursion depth exceeded


	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:661)
	at org.apache.spark.sql.execution.python.PythonArrowOutput$$anon$1.read(PythonArrowOutput.scala:101)
	at org.apache.spark.sql.execution.python.PythonArrowOutput$$anon$1.read(PythonArrowOutput.scala:50)
	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:614)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:489)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:754)
	at org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.hasNext(ArrowConverters.scala:147)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
	at scala.collection.Iterator.foreach(Iterator.scala:941)
	at scala.collection.Iterator.foreach$(Iterator.scala:941)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
	at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
	at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
	at scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
	at scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
	at scala.collection.AbstractIterator.to(Iterator.scala:1429)
	at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
	at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
	at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1429)
	at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
	at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
	at scala.collection.AbstractIterator.toArray(Iterator.scala:1429)
	at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$7(Dataset.scala:3727)
	at org.apache.spark.SparkContext.$anonfun$runJob$3(SparkContext.scala:2552)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:68)
	at org.apache.spark.scheduler.Task.doRunTask(Task.scala:148)
	at org.apache.spark.scheduler.Task.run(Task.scala:117)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$10(Executor.scala:732)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1643)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:735)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2766)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2713)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2707)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2707)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1256)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1256)
	at scala.Option.foreach(Option.scala:407)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1256)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2974)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2915)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2903)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:1029)
	at org.apache.spark.SparkContext.runJobInternal(SparkContext.scala:2458)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2441)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2553)
	at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$6(Dataset.scala:3725)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1643)
	at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$3(Dataset.scala:3729)
	at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$3$adapted(Dataset.scala:3695)
	at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3789)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$5(SQLExecution.scala:126)
	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:267)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$1(SQLExecution.scala:104)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:852)
	at org.apache.spark.sql.execution.SQLExecution$.withCustomExecutionEnv(SQLExecution.scala:77)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:217)
	at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3787)
	at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$2(Dataset.scala:3695)
	at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$2$adapted(Dataset.scala:3694)
	at org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$2(SocketAuthServer.scala:153)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1643)
	at org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$1(SocketAuthServer.scala:155)
	at org.apache.spark.security.SocketAuthServer$.$anonfun$serveToStream$1$adapted(SocketAuthServer.scala:150)
	at org.apache.spark.security.SocketFuncServer.handleConnection(SocketAuthServer.scala:124)
	at org.apache.spark.security.SocketFuncServer.handleConnection(SocketAuthServer.scala:117)
	at org.apache.spark.security.SocketAuthServer$$anon$1.$anonfun$run$4(SocketAuthServer.scala:70)
	at scala.util.Try$.apply(Try.scala:213)
	at org.apache.spark.security.SocketAuthServer$$anon$1.run(SocketAuthServer.scala:70)
Caused by: org.apache.spark.api.python.PythonException: 'pyspark.serializers.SerializationError: Caused by Traceback (most recent call last):
  File "/databricks/spark/python/pyspark/serializers.py", line 165, in _read_with_length
    return self.loads(obj)
  File "/databricks/spark/python/pyspark/serializers.py", line 469, in loads
    return pickle.loads(obj, encoding=encoding)
  File "/databricks/python/lib/python3.8/site-packages/faker/proxy.py", line 277, in __getattr__
    obj = getattr(self._proxy, name)
  File "/databricks/python/lib/python3.8/site-packages/faker/proxy.py", line 277, in __getattr__
    obj = getattr(self._proxy, name)
  File "/databricks/python/lib/python3.8/site-packages/faker/proxy.py", line 277, in __getattr__
    obj = getattr(self._proxy, name)
  [Previous line repeated 984 more times]
RecursionError: maximum recursion depth exceeded'. Full traceback below:
Traceback (most recent call last):
  File "/databricks/spark/python/pyspark/serializers.py", line 165, in _read_with_length
    return self.loads(obj)
  File "/databricks/spark/python/pyspark/serializers.py", line 469, in loads
    return pickle.loads(obj, encoding=encoding)
  File "/databricks/python/lib/python3.8/site-packages/faker/proxy.py", line 277, in __getattr__
    obj = getattr(self._proxy, name)
  File "/databricks/python/lib/python3.8/site-packages/faker/proxy.py", line 277, in __getattr__
    obj = getattr(self._proxy, name)
  File "/databricks/python/lib/python3.8/site-packages/faker/proxy.py", line 277, in __getattr__
    obj = getattr(self._proxy, name)
  [Previous line repeated 984 more times]
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

pyspark.serializers.SerializationError: Caused by Traceback (most recent call last):
  File "/databricks/spark/python/pyspark/serializers.py", line 165, in _read_with_length
    return self.loads(obj)
  File "/databricks/spark/python/pyspark/serializers.py", line 469, in loads
    return pickle.loads(obj, encoding=encoding)
  File "/databricks/python/lib/python3.8/site-packages/faker/proxy.py", line 277, in __getattr__
    obj = getattr(self._proxy, name)
  File "/databricks/python/lib/python3.8/site-packages/faker/proxy.py", line 277, in __getattr__
    obj = getattr(self._proxy, name)
  File "/databricks/python/lib/python3.8/site-packages/faker/proxy.py", line 277, in __getattr__
    obj = getattr(self._proxy, name)
  [Previous line repeated 984 more times]
RecursionError: maximum recursion depth exceeded


	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:661)
	at org.apache.spark.sql.execution.python.PythonArrowOutput$$anon$1.read(PythonArrowOutput.scala:101)
	at org.apache.spark.sql.execution.python.PythonArrowOutput$$anon$1.read(PythonArrowOutput.scala:50)
	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:614)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:489)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:754)
	at org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$1.hasNext(ArrowConverters.scala:147)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
	at scala.collection.Iterator.foreach(Iterator.scala:941)
	at scala.collection.Iterator.foreach$(Iterator.scala:941)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
	at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
	at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
	at scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
	at scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
	at scala.collection.AbstractIterator.to(Iterator.scala:1429)
	at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
	at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
	at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1429)
	at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
	at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
	at scala.collection.AbstractIterator.toArray(Iterator.scala:1429)
	at org.apache.spark.sql.Dataset.$anonfun$collectAsArrowToPython$7(Dataset.scala:3727)
	at org.apache.spark.SparkContext.$anonfun$runJob$3(SparkContext.scala:2552)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:68)
	at org.apache.spark.scheduler.Task.doRunTask(Task.scala:148)
	at org.apache.spark.scheduler.Task.run(Task.scala:117)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$10(Executor.scala:732)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1643)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:735)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

  warnings.warn(msg)
```


Discussion:

Resolving patch:
diff --git a/faker/proxy.py b/faker/proxy.py
--- a/faker/proxy.py
+++ b/faker/proxy.py
@@ -127,6 +127,9 @@ def __deepcopy__(self, memodict={}):
         }
         return result
 
+    def __setstate__(self, state):
+        self.__dict__.update(state)
+
     @property
     def unique(self):
         return self._unique_proxy
@@ -280,6 +283,16 @@ def __getattr__(self, name: str):
         else:
             raise TypeError("Accessing non-functions through .unique is not supported.")
 
+    def __getstate__(self):
+        # Copy the object's state from self.__dict__ which contains
+        # all our instance attributes. Always use the dict.copy()
+        # method to avoid modifying the original state.
+        state = self.__dict__.copy()
+        return state
+
+    def __setstate__(self, state):
+        self.__dict__.update(state)
+
     def _wrap(self, name, function):
         @functools.wraps(function)
         def wrapper(*args, **kwargs):


Test patch:
diff --git a/tests/test_proxy.py b/tests/test_proxy.py
--- a/tests/test_proxy.py
+++ b/tests/test_proxy.py
@@ -1,4 +1,5 @@
 import copy
+import pickle
 import random
 
 from collections import OrderedDict
@@ -396,3 +397,8 @@ def test_copy(self):
         fake2 = copy.deepcopy(fake)
         assert fake.locales == fake2.locales
         assert fake.locales is not fake2.locales
+
+    def test_pickle(self):
+        fake = Faker()
+        pickled = pickle.dumps(fake)
+        pickle.loads(pickled)