Issue 1:
Nikos Beis opened SPR-5476 and commented

When using many property files and a cacheSeconds value of bigger than 0 the lookup time increases rapidly. I saw a 15% performance boost in my application when putting all my properties in the same file. The problem is the nested for loops in resolveCode and resolveCodeWithoutArguments.

Issue Links:

#15133 ReloadableResourceBundleMessageSource locks properties hashmap and fails under load. ("duplicates")

#14948 Allow adding resources to ReloadableResourceBundleMessageSource

3 votes, 6 watchers

Issue 2:
Ronald R. DiFrango opened SPR-10500 and commented

The ReloadableResourceBundleMessageSource class locks the whole hashmap that holds properties and in our case is making the network call (after acquiring lock). This terribly slows down the property look-up. Under load it’s more significant as multiple threads could potentially try to load the files if there is a timeout.

Ideally we’d like a background thread load the properties after a timeout and put them into hashmap without locking as we are read-heavy and can tolerate stale entries.

Another option is to use a concurrent hashmap. Therefore the change for RRB should allow a backing map and whether to lock or not.

Here is the stack trace, notice the line in red is holding lock on a hashmap, which is the hashmap that holds properties:

"[ACTIVE] ExecuteThread: '150' for queue: 'weblogic.kernel.Default (self-tuning)'" id=8927 idx=0x2fc tid=30297 prio=5 alive, native_blocked, daemon
at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:32)[inlined]
at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)[inlined]
at java/net/SocketInputStream.read(SocketInputStream.java:129)[optimized]
at java/io/BufferedInputStream.fill(BufferedInputStream.java:218)
at java/io/BufferedInputStream.read1(BufferedInputStream.java:258)[optimized]
at java/io/BufferedInputStream.read(BufferedInputStream.java:317)[optimized]
^-- Holding lock: java/io/BufferedInputStream@0x10f0f4cc0[thin lock]
at weblogic/net/http/MessageHeader.isHTTP(MessageHeader.java:224)
at weblogic/net/http/MessageHeader.parseHeader(MessageHeader.java:148)[optimized]
at weblogic/net/http/HttpClient.parseHTTP(HttpClient.java:468)[optimized]
at weblogic/net/http/HttpURLConnection.getInputStream(HttpURLConnection.java:395)[inlined]
at weblogic/net/http/SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:37)[optimized]
^-- Holding lock: weblogic/net/http/SOAPHttpURLConnection@0x10f0f2550[biased lock]
at weblogic/net/http/HttpURLConnection.getResponseCode(HttpURLConnection.java:983)[optimized]
at org/springframework/core/io/AbstractFileResolvingResource.exists(AbstractFileResolvingResource.java:103)[optimized]
at com/capitalone/epf/context/support/ReloadableResourceBundleMessageSource.refreshProperties(ReloadableResourceBundleMessageSource.java:131)[optimized]
at org/springframework/context/support/ReloadableResourceBundleMessageSource.getProperties(ReloadableResourceBundleMessageSource.java:439)[inlined]
at org/springframework/context/support/ReloadableResourceBundleMessageSource.resolveCodeWithoutArguments(ReloadableResourceBundleMessageSource.java:272)[optimized]
^-- Holding lock: java/util/HashMap@0x10d49e228[fat lock]
at org/springframework/context/support/AbstractMessageSource.getMessageInternal(AbstractMessageSource.java:193)[inlined]
at org/springframework/context/support/AbstractMessageSource.getMessage(AbstractMessageSource.java:127)[optimized]
at org/springframework/context/support/AbstractApplicationContext.getMessage(AbstractApplicationContext.java:1210)[optimized]
at org/springframework/web/servlet/tags/MessageTag.resolveMessage(MessageTag.java:227)[inlined]
at org/springframework/web/servlet/tags/MessageTag.doStartTagInternal(MessageTag.java:165)[optimized]
at org/springframework/web/servlet/tags/RequestContextAwareTag.doStartTag(RequestContextAwareTag.java:79)[optimized]
at jsp_servlet/_web_45_inf/_views/__viewrequiredinformation._jsp__tag12(__viewrequiredinformation.java:1381)
at jsp_servlet/_web_45_inf/_views/__viewrequiredinformation._jspService(__viewrequiredinformation.java:441)
at weblogic/servlet/jsp/JspBase.service(JspBase.java:34)[optimized]
at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)[optimized]
at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:300)[optimized]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:183)[inlined]
at weblogic/servlet/internal/RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:523)[optimized]
at weblogic/servlet/internal/RequestDispatcherImpl.forward(RequestDispatcherImpl.java:253)[optimized]
at org/springframework/web/servlet/view/InternalResourceView.renderMergedOutputModel(InternalResourceView.java:238)[optimized]
at org/springframework/web/servlet/view/AbstractView.render(AbstractView.java:262)[optimized]
at org/springframework/web/servlet/DispatcherServlet.render(DispatcherServlet.java:1157)[optimized]
at org/springframework/web/servlet/DispatcherServlet.doDispatch(DispatcherServlet.java:927)[optimized]
at org/springframework/web/servlet/DispatcherServlet.doService(DispatcherServlet.java:827)[inlined]
at org/springframework/web/servlet/FrameworkServlet.processRequest(FrameworkServlet.java:882)[inlined]
at org/springframework/web/servlet/FrameworkServlet.doGet(FrameworkServlet.java:778)[optimized]
at javax/servlet/http/HttpServlet.service(HttpServlet.java:707)[optimized]
at javax/servlet/http/HttpServlet.service(HttpServlet.java:820)[optimized]
at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)[optimized]
at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:300)[optimized]
at weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at com/capitalone/epf/context/filters/EPFContextServletFilter.doFilter(EPFContextServletFilter.java:89)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[inlined]
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.wrapRun(Lweblogic/servlet/internal/ServletStubImpl;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)Ljava/lang/Object;(Unknown Source)[inlined]
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run()Ljava/lang/Object;(Unknown Source)[optimized]
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:120)[inlined]
at weblogic/servlet/internal/WebAppServletContext.securedExecute(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Z)V(Unknown Source)[inlined]
at weblogic/servlet/internal/WebAppServletContext.execute(Lweblogic/servlet/internal/ServletRequestImpl;Lweblogic/servlet/internal/ServletResponseImpl;)V(Unknown Source)[optimized]
at weblogic/servlet/internal/ServletRequestImpl.run()V(Unknown Source)[optimized]
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:209)[inlined]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)[optimized]
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

Now here are the lookup threads waiting on the lock [under load you could have literally hundreds as in our case]:

[ACTIVE] ExecuteThread: '146' for queue: 'weblogic.kernel.Default (self-tuning)'" id=8923 idx=0x300 tid=30008 prio=5 alive, blocked, native_blocked, daemon
-- Blocked trying to get lock: java/util/HashMap@0x10d49e228[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)[optimized]
at jrockit/vm/Locks.lockFat(Locks.java:1512)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
at org/springframework/context/support/ReloadableResourceBundleMessageSource.getProperties(ReloadableResourceBundleMessageSource.java:431)[inlined]
at org/springframework/context/support/ReloadableResourceBundleMessageSource.resolveCodeWithoutArguments(ReloadableResourceBundleMessageSource.java:272)[optimized]
at org/springframework/context/support/AbstractMessageSource.getMessageInternal(AbstractMessageSource.java:193)[inlined]
at org/springframework/context/support/AbstractMessageSource.getMessage(AbstractMessageSource.java:127)[optimized]
at org/springframework/context/support/AbstractApplicationContext.getMessage(AbstractApplicationContext.java:1210)[optimized]
at org/springframework/web/servlet/tags/MessageTag.resolveMessage(MessageTag.java:227)[inlined]
at org/springframework/web/servlet/tags/MessageTag.doStartTagInternal(MessageTag.java:165)[optimized]
at org/springframework/web/servlet/tags/RequestContextAwareTag.doStartTag(RequestContextAwareTag.java:79)[optimized]
at jsp_servlet/_web_45_inf/_views/__viewpaymentsalerts._jsp__tag34(__viewpaymentsalerts.java:2695)
at jsp_servlet/_web_45_inf/_views/__viewpaymentsalerts._jsp__tag30(__viewpaymentsalerts.java:2523)
at jsp_servlet/_web_45_inf/_views/__viewpaymentsalerts._jspService(__viewpaymentsalerts.java:601)
at weblogic/servlet/jsp/JspBase.service(JspBase.java:34)[optimized]
at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)[optimized]
at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:300)[optimized]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:183)[inlined]
at weblogic/servlet/internal/RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:523)[optimized]
at weblogic/servlet/internal/RequestDispatcherImpl.forward(RequestDispatcherImpl.java:253)[optimized]
at org/springframework/web/servlet/view/InternalResourceView.renderMergedOutputModel(InternalResourceView.java:238)[optimized]
at org/springframework/web/servlet/view/AbstractView.render(AbstractView.java:262)[optimized]
at org/springframework/web/servlet/DispatcherServlet.render(DispatcherServlet.java:1157)[optimized]
at org/springframework/web/servlet/DispatcherServlet.doDispatch(DispatcherServlet.java:927)[optimized]
at org/springframework/web/servlet/DispatcherServlet.doService(DispatcherServlet.java:827)[inlined]
at org/springframework/web/servlet/FrameworkServlet.processRequest(FrameworkServlet.java:882)[inlined]
at org/springframework/web/servlet/FrameworkServlet.doGet(FrameworkServlet.java:778)[optimized]
at javax/servlet/http/HttpServlet.service(HttpServlet.java:707)[optimized]
at javax/servlet/http/HttpServlet.service(HttpServlet.java:820)[optimized]
at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)[optimized]
at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:300)[optimized]
at weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at com/capitalone/epf/context/filters/EPFContextServletFilter.doFilter(EPFContextServletFilter.java:89)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[inlined]
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.wrapRun(Lweblogic/servlet/internal/ServletStubImpl;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)Ljava/lang/Object;(Unknown Source)[inlined]
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run()Ljava/lang/Object;(Unknown Source)[optimized]
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:120)[inlined]
at weblogic/servlet/internal/WebAppServletContext.securedExecute(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Z)V(Unknown Source)[inlined]
at weblogic/servlet/internal/WebAppServletContext.execute(Lweblogic/servlet/internal/ServletRequestImpl;Lweblogic/servlet/internal/ServletResponseImpl;)V(Unknown Source)[optimized]
at weblogic/servlet/internal/ServletRequestImpl.run()V(Unknown Source)[optimized]
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:209)[inlined]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)[optimized]
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

Affects: 3.1 GA

Attachments:

ReloadableResourceBundleMessageSource.java (26.71 kB)

Issue Links:

#10149 ReloadableResourceBundleMessageSource performance issue when using many resource files ("is duplicated by")

#19152 Method getMergedProperties in ReloadableResourceBundleMessageSource does not set fileTimestamp

#20782 Remove synchronization from ResourceBundleMessageSource

Referenced from: commits a000dd7

7 votes, 6 watchers