Repository: brootware/PyRedactKit (language: python)

Issue:
Explore into async or multi processing for faster file processing
<!-- NOTE: Please maintain all sections, otherwise the issue will be automatically closed :) -->

## Checklist

<!-- Please complete the following list of tasks, and then check it by changing the "[ ]" to "[x]" -->
- [x] There are no similar reports on [existing issues](https://github.com/brootware/PyMouseBot/issues?q=is%3Aissue) (including closed ones).
- [x] I was in the `master` branch of the latest code.

## Is your feature request related to a problem? Please describe

<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
For large multiple files, the process time takes (n number of files * time).

## Describe the solution you'd like

<!-- A clear and concise description of what you want to happen. -->
Explore into either async or multi processing for handling multiple files in parallel.

## Describe alternatives you've considered

<!-- A clear and concise description of any alternative solutions or features you've considered. -->
Threading is one other option

## Additional context

<!-- Add any other context or screenshots about the feature request here. -->



Discussion / hints:
Update : Async does not really help as the task is not I/O bound but CPU bound after timing the program run time. Will have to use multiprocessing for multiple files redaction.

```bash
python3 pyredactkit.py ip_test.txt  39.31s user 0.16s system 100% cpu 39.412 total
```
More update after running benchmarks. Reading and Writing files to disk is already non blocking. The "concurrency" only comes in handy when you're doing requests over the network.

![image](https://user-images.githubusercontent.com/7734956/181907750-684622cc-99f0-4282-a262-35cbbc940b72.png)

For running 5 text files

```
With threading
poetry run prk logtest  39.58s user 1.15s system 97% cpu 41.678 total

Without threading
poetry run prk logtest  20.05s user 0.43s system 93% cpu 21.897 total
```

Fix patch:
diff --git a/pyredactkit/identifiers.py b/pyredactkit/identifiers.py
index 3f37adc..950a800 100644
--- a/pyredactkit/identifiers.py
+++ b/pyredactkit/identifiers.py
@@ -28,7 +28,7 @@ class Identifier:
             "type": ("cc", "creditcard")
         },
         {
-            "pattern": r"[STFG]\d{7}[A-Z]",
+            "pattern": r"[STFGM]\d{7}[A-Z]",
             "type": ("nric", "fin", "ic")
         },
         {


Test patch:
diff --git a/tests/test_runner.py b/tests/test_runner.py
index 13f26fc..f383888 100644
--- a/tests/test_runner.py
+++ b/tests/test_runner.py
@@ -46,8 +46,9 @@ def test_api_identify_sensitive_data(mocker_text_file, tmp_path):
     S0000212Q
     F2120212E
     S0021001P
+    M1234567A
     """
-    test_data = ['102.23.5.1', '10.10.10.1', '71.159.188.33', '81.141.167.45', '165.65.59.139', '64.248.67.225', 'https://tech.gov.sg', 'harold@mail.com', 'mail.com', '102.23.5.1', '10.10.10.1', '71.159.188.33', '81.141.167.45', '165.65.59.139', '64.248.67.225', 'G0022121F', 'F2121200F', 'G1021022E', 'S1022221L', 'G1222221C', 'S0000212Q', 'F2120212E', 'S0021001P']
+    test_data = ['102.23.5.1', '10.10.10.1', '71.159.188.33', '81.141.167.45', '165.65.59.139', '64.248.67.225', 'https://tech.gov.sg', 'harold@mail.com', 'mail.com', '102.23.5.1', '10.10.10.1', '71.159.188.33', '81.141.167.45', '165.65.59.139', '64.248.67.225', 'G0022121F', 'F2121200F', 'G1021022E', 'S1022221L', 'G1222221C', 'S0000212Q', 'F2120212E', 'S0021001P', 'M1234567A']
 
     assert Runner.api_identify_sensitive_data(test_string) == test_data, "api_identify_sensitive_data function should return a list of sensitive data"
     assert Runner.api_identify_sensitive_data('This is a test string') == [], "api_identify_sensitive_data function should return an empty list"
\ No newline at end of file