crawljax/crawljax — pull request #228 by midlife83 (2013-04-03 18:24:21+00:00)
Title: ISSUE #102 & ISSUE #75

Team L2C3 - EECE 310

Re-opening the pull request: removed unnecessary Javadoc comments and used the formmater as requested. Below is the original pull request message.

Work done for ISSUE# 102. The issue is not resolved because the problem could not be recreated whiling running example code on 8 different websites for more than 2 times per website.
Summary of Work:
1.Major overhaul on CrawlQueue.java. Instead of extending Stack, it now just uses a list to manually control elements. Implemented all methods for the BlockingQueue interface in Thread-safe manner.
2.Created a JUNIT test, CrawlQueueTest.java for CrawlQueue.java. Tests all the methods. Non-trivial methods are tested using multiple threads to check for thread-safeness.
3.Modified the constructor of CrawlerExecutor,java. Originally it was unbounded but now is bounded with RejectedExecutionHandler set as CallerRunsPolicy which makes the main thread to run the task when the queue is full (instead of throwing an exception as in the original setting). Unbounded queue = source of out of memory error. Increased the default number of threads.
4.Modified CrawlerExecutorTest.java. Added more threads to test CrawlerExecutor.java and added delays. I just pulled from the original repo and it seems that somebody has already added Thread.sleep(1). Not sure this is done after I originally submitted a pull request (which I closed promptly to work more on those files and resubmit. More explanation provided below:

While working on the issue#102, found the reason for issue#75. The reason that CrawlerExecutorTest.java fails sometimes without any apparent reason is that the OS scheduler may cause the order of execution differ from the order of statements in the code. Slight delay between ThreadPoolExecutor.execute() is needed for this test to function correctly. Depending on how the OS assigns JAVA threads to cores (or OS threads) and the load of the cores at the moment, the speed of execution of the methods could differ from the actual order of calling them. As a result, sometimes this test fails when executor.execute(t2) is actually executed before executor.execute(t1). There are only two ways to fix this. First is to put a small delay (for example, Thread.sleep(SLEEPTIME)) between method callings. Or assign only one core to JAVA SE process on Windows Task Manager (or equivalent features of other OS).

Maven integration test has been passed.