RequestPolicy/requestpolicy — Issue #192: Import Hangs on ~500K Policies
Author: jsamuel
Created: 2011-12-22 04:10:54+00:00
State: open
Labels: extension

Given the patches presented in [ticket:190#comment:1], tried to expand a single regex of the form:

{{{http://leaf[0-9]{2}.parent[0-9].node.example.com}}}

This yields:

~1,000 Policies for {{{http://example.com}}} to any leaf.
~500,000 Policies for any leaf to any leaf

Yes, '''that''' many policies can be represented by a single regular expression -- and the only reason to expand one as such, is that #8 is still pending...

Fortunately, I had imported as TEMPORARY to ensure that the policies would not persist if an OOM condition occurred.

Importing ~1,000 policies worked without noticeable impact, even when queried during browsing; however, importing ~500,000 policies completely hung firefox during import.

Not quite patient enough to wait and see if an OOM would eventually result, if it would crash prior, or just ''really'' slow to a crawl, I killed the process after several minutes of swap file induced disk thrashing.

I anticipate most users would never reach even 1,000 policies unless importing, such as if request policy was used as some sort of organizational policy with a comprehensive policy set for the organization. Still, there are better ways to implement that as well.

Nevertheless, the import dialog should probably check the length of the file before trying to blindly import the file and warn the user if the file contains more than '''X''' policies (really just lines, each are potential policies -- blanks are ignored and there are a few headers). The user should be warned to the effect that such a large import can noticeably impact the browsing experience, and has a potential for detrimental effects, eg an OOM condition...

Now we just need to identify what a reasonable '''X''' is... 10,000?