Package: grip-mbmasuda
generic-rate-limited-item-processorWhat this package containsThis package contains the code that will allow you to create a queue or stack
of items to process. The default behavior is a queue, but the internal implementation
uses a deque so you can easily subclass this code and override the methods that
push and pop items to change the behavior from a queue to a stack.Basically, there is a class calledGenericRateLimitedItemProcessorthat is a
thread and when the thread is started, the items in the deque of things to process
are popped off one at a time and whatever they are supposed to do gets started
when theGenericRateLimitedItemProcessorpops the next item and calls the item'sstart()method.Each item in theGenericRateLimitedItemProcessor's deque must implement astart()method, but this is checked generically via a metaclass implementation of__instancecheck__so that any callablestartattribute anywhere in the MRO will
satisfy this requirement. Thus, items in theGenericRateLimitedItemProcessor's deque
can be of mixed types and include threads and simple non-threads as well.Because subclasses ofthreading.Threadwill obviously already have astart()method
since that is how the thread'srun()implementation gets started, that is the easy
way of making items to be processed. However, if you don't need anything complicated
and the overhead of making a bunch of threads is unneccessary, you can implement the
simnplest class possible, like this:classSimplestPossibleItemThatCanBeProcessed:defstart(self):passand replace thepasswith whatever you need to happen.Installation$python3-mpipinstallgrip-mbmasudaExample usage (not a comprehensive tutorial)fromsrc.grip.gripimportGenericRateLimitedItemProcessorclassNumberItem:defstart(self):return42classSayHiItem:defstart(self):return'hi!'# make some items; you can mix and match and have different types of items processed by# the same processoritems=[NumberItem()forxinrange(5)]+[SayHiItem()forxinrange(5)]# create two item processors that will process the same data# the only requirement about the items is that they all have a callable start() method# this item processor will process as fast as possibleitem_processor_that_processes_as_fast_as_possible=GenericRateLimitedItemProcessor(iterable=items)# this one's processing rate is limited to 1000 items in 1.25 secondsitem_processor_with_rate_limited_to_1000_items_in_1.25_seconds=GenericRateLimitedItemProcessor(iterable=items,num_items=1000,num_seconds=1.25)# start processing!item_processor_that_processes_as_fast_as_possible.start()item_processor_with_rate_limited_to_1000_items_in_1.25_seconds.start()# wait for all items's start() to be called# if you want your items to be processed asynchronosuly, you should have your items# be a subclass of threading.Thread, or the processing they do should be extremely# minimal and not block, otherwise calling start() on the subsequent items in the item# processor could be delayed and your item processor will not appear to be processing# at the rate you think it should be processing atitem_processor_that_processes_as_fast_as_possible.join()item_processor_with_rate_limited_to_1000_items_in_1.25_seconds.join()# the items where calling start() raised an exception are collected in this dequeproblem_items=item_processor_that_processes_as_fast_as_possible.unsuccessfully_started# the items where calling start() was successful and did not raise are collected in this dequegood_items=item_processor_that_processes_as_fast_as_possible.successfully_started# just because an item ends up in the successfully_started deque does not mean your item's# processing code exited without errors; it only means that the item's start() was called# without raising an excaption and you are responsible for going through the items in the# successfully_started deque to check status. For example, if the item is a thread whose# run() has code that makes an HTTP request, then you are responsible for checking the# request's response to ensure it did not come back with an HTTP 4xx or 5xx (or whatever# you need it to do)Testing$ py.test -vvvv
====================================================== test session starts ======================================================
platform darwin -- Python 3.9.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- <redacted>
cachedir: <redacted>
rootdir: <redacted>
collected 26 items

test/test_everything.py::TestPorcelain::test_make_new_instances PASSED                                                    [  3%]
test/test_everything.py::TestPorcelain::test_isinstance_works_correctly PASSED                                            [  7%]
test/test_everything.py::TestPorcelain::test_simplest_possible_item_that_can_be_processed PASSED                          [ 11%]
test/test_everything.py::TestPorcelain::test_mixed_and_matched_items PASSED                                               [ 15%]
test/test_everything.py::TestPorcelain::test_valid_items_can_be_processed PASSED                                          [ 19%]
test/test_everything.py::TestPorcelain::test_rate_limiting_works[100-1] PASSED                                            [ 23%]
test/test_everything.py::TestPorcelain::test_rate_limiting_works[3-1.98324729923487] PASSED                               [ 26%]
test/test_everything.py::TestPorcelain::test_rate_limiting_works[2344-0.3] PASSED                                         [ 30%]
test/test_everything.py::TestPlumbing::test_make_new_instance[iterable_data0-None] PASSED                                 [ 34%]
test/test_everything.py::TestPlumbing::test_make_new_instance[iterable_data1-4] PASSED                                    [ 38%]
test/test_everything.py::TestPlumbing::test_make_new_instance[iterable_data2-None] PASSED                                 [ 42%]
test/test_everything.py::TestPlumbing::test_make_new_instance[iterable_data3-7] PASSED                                    [ 46%]
test/test_everything.py::TestPlumbing::test_make_new_instance[iterable_data4-3] PASSED                                    [ 50%]
test/test_everything.py::TestPlumbing::test_make_new_instance[abcdefg-None] PASSED                                        [ 53%]
test/test_everything.py::TestPlumbing::test_get_next_item[iterable_data0-None] PASSED                                     [ 57%]
test/test_everything.py::TestPlumbing::test_get_next_item[iterable_data1-4] PASSED                                        [ 61%]
test/test_everything.py::TestPlumbing::test_get_next_item[iterable_data2-None] PASSED                                     [ 65%]
test/test_everything.py::TestPlumbing::test_get_next_item[iterable_data3-7] PASSED                                        [ 69%]
test/test_everything.py::TestPlumbing::test_get_next_item[iterable_data4-3] PASSED                                        [ 73%]
test/test_everything.py::TestPlumbing::test_get_next_item[abcdefg-None] PASSED                                            [ 76%]
test/test_everything.py::TestPlumbing::test_append_item[iterable_data0-None] PASSED                                       [ 80%]
test/test_everything.py::TestPlumbing::test_append_item[iterable_data1-4] PASSED                                          [ 84%]
test/test_everything.py::TestPlumbing::test_append_item[iterable_data2-None] PASSED                                       [ 88%]
test/test_everything.py::TestPlumbing::test_append_item[iterable_data3-7] PASSED                                          [ 92%]
test/test_everything.py::TestPlumbing::test_append_item[iterable_data4-3] PASSED                                          [ 96%]
test/test_everything.py::TestPlumbing::test_append_item[abcdefg-None] PASSED                                              [100%]

====================================================== 26 passed in 3.95s =======================================================How to get in touchPlease email me atmbmasuda.github@gmail.com