Tools suggestion for testing with RPC api

username: iffiX
cooked: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> This question is not that closely related to pytorch, but it is really hard to find related resources on stackoverflow etc. My framework (mentioned in <a href="https://discuss.pytorch.org/t/about-distributed-autograd-and-optimizer/84047/6">last post</a>) development have finally reached a critical stage, I am testing the distributed part now.</p>
<p>So do you know any python libraries such as <a href="https://github.com/NetSys/demi" rel="nofollow noopener">DEMi</a>, <a href="https://github.com/nccgroup/fuzzowski" rel="nofollow noopener">fuzzowski</a> which can simulate a connection layer and let users <strong>send &amp; recv</strong>, <strong>shift, reorder &amp; delay</strong>, <strong>poke &amp; inspect</strong> messages? Since I need to simulate the RPC layer, these two libraries metioned above are not appropriate for this task. What are the development team of pytorch using to test your rpc service? Any idea would be great.</p>
<p>BTW, my library is hosted at <a href="https://github.com/iffiX/machin" rel="nofollow noopener">https://github.com/iffiX/machin</a> <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"> Come and have a look! the tutorial is not complete though, but api documentation and tests for most core functions should be close to complete now, will release the first milestone soon.</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/iffix">@iffiX</a>, thanks a lot for sharing the exciting project!! We will certainly study and share with other team members. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Regarding tests for delay/drop/retry messages, <a class="mention" href="/u/osalpekar">@osalpekar</a> implemented a <a href="https://github.com/pytorch/pytorch/blob/1e64bf4c40ef82d6bc3dcc42b3874353f7632be0/torch/csrc/distributed/rpc/testing/faulty_process_group_agent.h" rel="noopener nofollow ugc">faulty agent</a> for that purpose, and the majority of faulty tests can be found <a href="https://github.com/pytorch/pytorch/blob/1e64bf4c40ef82d6bc3dcc42b3874353f7632be0/torch/testing/_internal/distributed/rpc/rpc_test.py#L3398" rel="noopener nofollow ugc">here</a>.</p>
<blockquote>
<p>So do you know any python libraries such as <a href="https://github.com/NetSys/demi" rel="noopener nofollow ugc">DEMi</a>, <a href="https://github.com/nccgroup/fuzzowski" rel="noopener nofollow ugc">fuzzowski</a> which can simulate a connection layer and let users  <strong>send &amp; recv</strong> ,  <strong>shift, reorder &amp; delay</strong> ,  <strong>poke &amp; inspect</strong>  messages? Since I need to simulate the RPC layer, these two libraries metioned above are not appropriate for this task.</p>
</blockquote>
<p>If this is just for simulating RPCs, does mock work in this case? Sth like <a href="https://github.com/pytorch/pytorch/pull/39728/files" rel="noopener nofollow ugc">this</a>.</p>
<p>Also cc TensorPipe expert <a class="mention" href="/u/lcw">@lcw</a>, do you know any good tools for this purpose?</p>

username: iffiX
cooked: <p>Mock is not sufficient, because I would like to implement tests based on fuzzy testing for <code>machin.parallel.distributed.election.ElectionGroupStableBase</code> and other higher level modules based on this core. Common white box testing and black box testing are not good enough for testing distributed core functions, so my testing idea comes from this <a href="https://github.com/asatarin/testing-distributed-systems" rel="nofollow noopener">github repo</a> and the blog post from its “Fuzzing” section <a href="https://colin-scott.github.io/blog/2015/10/07/fuzzing-raft-for-fun-and-profit/" rel="nofollow noopener">blog</a>.</p>
<p>I think I will try to implement a simple framework myself today, maybe generate fuzzy data with <a href="https://github.com/jtpereyda/boofuzz" rel="nofollow noopener">boofuzz</a> and write a rpc simulation layer. I am still looking forward to other ideas of your team! <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>

username: iffiX
cooked: <p>I made a rpc mocker &amp; fuzzer, should be sufficient for now.</p>