Project: bamboo
Issue BAM-9941: ActiveMQ JMS data file no longer found causing builds not able to launch
{noformat:title=BEAC server log JMS summary snipppet} 2011-10-09 18:34:29,222 ERROR [BrokerService[bamboo] Task] [Queue] Failed to page in more queue messages java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file /opt/j2ee/domains/bamboo.atlassian.com/confluence/webapps/atlassian-bamboo/data/current/jms-store/bamboo/KahaDB/db-1782.log at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113) at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157) at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1677) at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1897) at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1424) at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file /opt/j2ee/domains/bamboo.atlassian.com/confluence/webapps/atlassian-bamboo/data/current/jms-store/bamboo/KahaDB/db-1782.log at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265) at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110) ... 9 more Caused by: java.io.IOException: Could not locate data file /opt/j2ee/domains/bamboo.atlassian.com/confluence/webapps/atlassian-bamboo/data/current/jms-store/bamboo/KahaDB/db-1782.log at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345) at org.apache.kahadb.journal.Journal.read(Journal.java:592) at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786) at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956) at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494) at org.apache.kahadb.page.Transaction.execute(Transaction.java:728) at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485) at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88) at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262) ... 10 more 2011-10-09 18:34:29,351 ERROR [BrokerService[bamboo] Task] [Journal] Looking for key 1782 but not found in fileMap: {1783=db-1783.log number = 1783 , length = 33554836, 1786=db-1786.log number = 1786 , length = 34109378, 1787=db-1787.log number = 1787 , length = 33556132, 1280=db-1280.log number = 1280 , length = 33556132, 1784=db-1784.log number = 1784 , length = 33554712, 1790=db-1790.log number = 1790 , length = 33606358, 1791=db-1791.log number = 1791 , length = 35664711, 1788=db-1788.log number = 1788 , length = 33581527, 1789=db-1789.log number = 1789 , length = 33557879, 1806=db-1806.log number = 1806 , length = 33592825, 1807=db-1807.log number = 1807 , length = 16003513, 1804=db-1804.log number = 1804 , length = 33635071, 1805=db-1805.log number = 1805 , length = 33554607, 1802=db-1802.log number = 1802 , length = 33580009, 1803=db-1803.log number = 1803 , length = 33554674, 1800=db-1800.log number = 1800 , length = 33567294, 1801=db-1801.log number = 1801 , length = 33819685, 1798=db-1798.log number = 1798 , length = 33971508, 1279=db-1279.log number = 1279 , length = 33554625, 1799=db-1799.log number = 1799 , length = 33630766, 1796=db-1796.log number = 1796 , length = 33651278, 1797=db-1797.log number = 1797 , length = 45623364, 1794=db-1794.log number = 1794 , length = 33666339, 1795=db-1795.log number = 1795 , length = 34512258, 1792=db-1792.log number = 1792 , length = 35143715, 1793=db-1793.log number = 1793 , length = 33776959} {noformat} {noformat:title=remote agents unable to send heartbeats} 2011-10-10 09:55:43,655 INFO [QuartzScheduler_Worker-9] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 13 seconds ago, dropping the current heartbeat... 2011-10-10 10:05:43,655 INFO [QuartzScheduler_Worker-6] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 7 seconds ago, dropping the current heartbeat.. {noformat} Likely caused by these ActiveMQ Issues: https://issues.apache.org/jira/browse/AMQ-3310 or: https://issues.apache.org/jira/browse/AMQ-3470 Reported internally in: https://extranet.atlassian.com/jira/browse/BUILDENG-649 https://extranet.atlassian.com/jira/browse/BUILDENG-651 The JMS spam caused the server logs to roll over so I was unable to capture the beginning of the problem. Confirmed server side that {{db-1782.log}} in the JMS repo was not present
Story points: 2