GitHub PushEvent at 2014-12-01 10:12:52
Repository: dcache
Actor: kschwank
Payload: {"shas":[["ef00af132cbc2cb0c801d4a4be2022580ca9f5b7","6cfe06726305b12ef8f52d7564ec2018cefd2794@desy.de","update schema version from 3.0 to 3.1","Karsten Schwank",true],["e76e54f94e99634fcb0b99cf2ceae4aa67714f97","e9ea248744a122662c709a2d6819bccccd67c1f5@otfrid.fnal.gov","datanucleus: use persistence units\n\nBased on the 2.11 release notes, a patch to embed the alarms service in dCacheDomain of system-test\nwas committed. The release notes claimed that there no longer was a requirement to run it in a separate domain.\n\nEmbedding it however broke the tests because the EnhancementHelper.registerClass() call for the XML database (alarms)\nrejects other classes using the datastore identity, even though not destined for storage in that database.\n\nThe issue here, to quote the author of the code, is as follows:\n\n\"JDO comes with an auto-notify feature that will inform ALL interested parties when an enhanced class is loaded (so they can load up metadata for it). Since you have multiple PMF's then the MetaDataManager of each PMF are informed and they try to load it (since it is simply an enhanced class - can be used with any persistence environment that supports JDO). Since one PMF is for XML and it only accepts particular metadata then it rejects that class (for that MetaDataManager) by exception. DN 4.1 code simply logs a warning (instead of exception) that the class is not available for that PMF (so the user can fix their metadata if they want it to be available there).\"\n\nFor fuller discussion, see http://www.datanucleus.org/servlet/forum/viewthread_thread,7815.\n\nWhile an update to 4.1 will remove the failure, the better solution is the one we have adopted:\nthe suggested partitioning of all enhanced classes by persistence units (already used for the pinmanager).\n\nAlso makes the 'off' url for alarms a non-empty string; otherwise an NPE occurs during injection.\n\nTarget: trunk\nPatch: https://rb.dcache.org/r/7517\nRequest: 2.11\nRequire-book: no\nRequire-notes: yes (in case 2.11 has been used already)\nAcked-by: Gerd\n\nRELEASE NOTES:  Fixes error when alarms XML database and TransferManager share the same domain.","alrossi",true],["c59f61eb89d04948f7e8172b15cbd8288778533c","6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","spacemanager: Fix various error recovery scenarios\n\nThe patch fixes several related issues in space manager:\n\n- The code used to delete a file reservation when the mover reported\n  a failure. However whether such a file is actually deleted depends\n  on what the door does. If the door deletes it, the space manager\n  will be notified and the file can be deleted at that point.\n\n  The special case that exists is if the mover failure was caused by the\n  pool being unable to register the location in the name space. When\n  such a file is deleted it would only get a name space delete notification,\n  and no pool remove file notification. To account for this case, the\n  pool leaves a file for which the mover reported an error in the\n  transferring state. If the name space entry is deleted, such an entry\n  is removed. If the file is not deleted, the other fixes of this\n  patch will address it eventually.\n\n- The code used to assume that a file that was in transferring\n  would either get a DoorTransferFinished notification or be deleted.\n  There is however no guarantee that this happens for non-srm uploads,\n  such as when the door doesn't delete on failed upload. The client\n  may even think the transfer is successfull if no control channel is\n  used (eg http and xrootd). Such a file is perfectly readable, but\n  stayed in transferring in space manager.\n\n  This patch addresses this problem by extending the background check\n  of files that have stayed in transferring for more than 24 hours:\n  rather than just deleting reservations for deleted files, the code\n  now also detects flushed and completed files and updates the reservation\n  accordingly.\n\n- The patch changes how we react to notifications that a name space entry\n  is deleted: Before, we deleted the reservation if the file was marked\n  flushed. For other states we assumed we would eventually get a pool\n  remove file notification: However if the pool failed to register the\n  location, we will never get such a notification. This scenario is\n  quite likely if the name space entry is deleted during upload.\n\n  The patch addresses this problem by deleting a transferring reservation on\n  the name space delete notification. The downside is that we have some\n  space allocated for the file on the pool that isn't explicitly linked\n  to a reservation in space manager, but since the name space entry got\n  deleted, the file on the pool will soon get deleted too.\n\n- The patch also refines how we react to notifications for flushed files: if the\n  reservation was in transferring, it is now considered complete.  Before the\n  flush notification was simply ignored. This could both happen for lost\n  DoorTransferFinished notification, of if the flush notification managed to\n  overtake the DoorTransferFinished notification (unlikely).\n\nTarget: trunk\nRequest: 2.11\nRequest: 2.10\nRequest: 2.9\nRequire-notes: yes\nRequire-book: yes\nAcked-by: Dmitry Litvintsev <litvinse@fnal.gov>\nPatch: https://rb.dcache.org/r/7519/","Gerd Behrmann",true],["c36188ca9a477b4eaff3724bd9e78c635faf835c","6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","pnfsmanager: Propagate cleaner errors to RemoteNameSpaceProvider\n\nThe NameSpaceProvider interface is not asynchronous and cleaning\na pool location through this interface should propagate errors\nback to the client.\n\nTarget: trunk\nRequire-notes: no\nRequire-book: no\nAcked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>\nAcked-by: Paul Millar <paul.millar@desy.de>\nPatch: https://rb.dcache.org/r/7501/","Gerd Behrmann",true],["9317083dcb67512cb72636b4802e152db55cd7fa","6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","info: Reduce delay between messages\n\nThe info service takes a long time to populate its state after a restart, in\nparticular on production scale systems.\n\nThis patch reduces the delay between queries. The overall frequency is not\nincreased, but the delay between two queries is reduced to 100 ms (from 2, 5,\nor 10 seconds, depending on the query). As these queries are relatively\nlight-weight, 100 ms should be plenty to keep the message passing overhead\ndown.\n\nI request backport to 2.10 as we always have issues with our Nagios probes\nafter a restart.\n\nTarget: trunk\nRequest: 2.11\nRequest: 2.10\nRequire-notes: yes\nRequire-book: no\nAcked-by: Paul Millar <paul.millar@desy.de>\nPatch: https://rb.dcache.org/r/7535/","Gerd Behrmann",true],["e9738abc62936db005c8962ee4b568523c129d5b","6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","httpd: Drop info pages from webadmin\n\nThe info pages exist because originally webadmin was developed as a complete\nreplacement for httpd, meaning everything httpd could do, webadmin had to do\ntoo, even if that functionallity wasn't really a part of the admin tool.\n\nThe info for info-provider is just as easy to expose through the classic\ninfo link - info-provider was actually never updated to use webadmin.\n\nEven the UI in webadmin for accessing the info xml is inferior to just\nload the old page in the browser.\n\nThis patch removes both the Info and InfoXml pages from webadmin.\n\nTarget: trunk\nRequire-notes: yes\nRequire-book: yes\nAcked-by: Albert Rossi <arossi@fnal.gov>\nPatch: https://rb.dcache.org/r/7536/","Gerd Behrmann",true],["61dfae7938bbb171d8ac777fb182ef512ff77f5f","6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","pool: Fix ISE when flushing deleted file\n\nPrior to flushing a file, the pool checks if the name space entry still exists. If\nnot, it deletes the replica rather than flushing it to tape. Release 2.10.7 introduced\na regression that causes this code path to throw an ISE:\n\n11:41:01 [pool-28-thread-1] [1f45dec7-f415-4dea-80cb-57c825b51486] Flush of 000084C99C4AC70E4CC38279851A856FABB8 failed with: CacheException(rc=10001;msg=No such file or directory: 000084C99C4AC70E4CC38279851A856FABB8).\n11:41:01 [pool-12-thread-2] [1f45dec7-f415-4dea-80cb-57c825b51486] remove entry for: 000084C99C4AC70E4CC38279851A856FABB8\n11:41:02 [pool-10-thread-1] [door:GFTP-Gerds-MacBook-Pro-5-AAUHz0KyLFg GFTP-Gerds-MacBook-Pro-5-AAUHz0KyLFg PoolAcceptFile 000084C99C4AC70E4CC38279851A856FABB8] Unexpected failure during state change notificationjava.lang.IllegalStateException: Handle is closed\n    at org.dcache.pool.repository.v5.ReadHandleImpl.close(ReadHandleImpl.java:48) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]\n    at org.dcache.pool.nearline.NearlineStorageHandler$FlushRequestImpl.failed(NearlineStorageHandler.java:628) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]\n    at org.dcache.pool.nearline.AbstractBlockingNearlineStorage$Task.cancel(AbstractBlockingNearlineStorage.java:200) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]\n    at org.dcache.pool.nearline.AbstractBlockingNearlineStorage.cancel(AbstractBlockingNearlineStorage.java:71) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]\n    at org.dcache.pool.nearline.NearlineStorageHandler$AbstractRequest.cancel(NearlineStorageHandler.java:365) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]\n    at org.dcache.pool.nearline.NearlineStorageHandler$AbstractRequestContainer.cancel(NearlineStorageHandler.java:446) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]\n    at org.dcache.pool.nearline.NearlineStorageHandler.stateChanged(NearlineStorageHandler.java:279) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]\n    at org.dcache.pool.repository.v5.StateChangeListeners$1.run(StateChangeListeners.java:68) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]\n    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_25]\n    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_25]\n    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]\n\n11:41:02 [pool-12-thread-2] [1f45dec7-f415-4dea-80cb-57c825b51486] File not found in name space; removed 000084C99C4AC70E4CC38279851A856FABB8.\n\nThis is caused by the pre-flush check deleting the replica followed by failing the flush\nrequest. Deleting the replica subsequently causes a notification to be sent to the HSM\nsubsystem, which causes the same flush request to be cancelled. The script driver reports\na second failure, thus causing the replica handler to be closed twice - hence the ISE.\n\nThe patch fixes the regression.\n\nThe log message is also cleaned up to better state what happened:\n\n12:07:39 [pool-28-thread-1] [84967682-8766-4e66-80c3-3199f047b6b2] Flush of 00005466F5DC5AEA4263B3FA8E6EECCCF045 failed with: CacheException(rc=10001;msg=File not found in name space during pre-flush check.).\n12:07:39 [pool-12-thread-2] [84967682-8766-4e66-80c3-3199f047b6b2] remove entry for: 00005466F5DC5AEA4263B3FA8E6EECCCF045\n\nTarget: trunk\nRequest: 2.11\nRequest: 2.10\nRequire-notes: yes\nRequire-book: no\nAcked-by: Karsten Schwank <karsten.schwank@desy.de>\nAcked-by: Paul Millar <paul.millar@desy.de>\nPatch: https://rb.dcache.org/r/7506/","Gerd Behrmann",true],["7231c6557e102130501f59af19cdd224822ba186","d922616d71ca96f778c63ea49015b141afa0a2ae@desy.de","pnfsmanager: update atime on get attribute if desired\n\nPnfsManager still controls the time interval during which updates are ignored.\n\nAcked-by: Gerd Behrmann\nTarget: master\nRequire-book: yes\nRequire-notes: yes","Tigran Mkrtchyan",true],["681bc2a659794e2c70d56496f4c9d5fa916db95c","d922616d71ca96f778c63ea49015b141afa0a2ae@desy.de","chimera: make Stat aware of defined attributes\n\nupdate FsSqlDriver to generate SQL statement for required\nattributes only.\n\nAcked-by: Gerd Behrmann\nTarget: master\nRequire-book: no\nRequire-notes: no","Tigran Mkrtchyan",true],["239b1de4ff6cba8b8904d58451256425608fe6da","d922616d71ca96f778c63ea49015b141afa0a2ae@desy.de","chimera-provider: do not get file attributes prior update\n\nAcked-by: Gerd Behrmann\nTarget: master\nRequire-book: no\nRequire-notes: yes","Tigran Mkrtchyan",true],["7c43c2ad6a6d6209625ee78d6745a5f8c531d7a3","d922616d71ca96f778c63ea49015b141afa0a2ae@desy.de","pool: reorder ip addresses returned to NFS client\n\naccording to NFS spec pool client can select a best matching IP\naddress from the list available IP addresses of the pool.\n\nNevertheless, current client implementation in RHEL-6 takes first\none from the list and, some times, makes a wrong choice for mutihomed\npools.\n\nThis change will put 'expected correct' interface as a first element\nof the returned list.\n\nTicket: #8404\nAcked-by: Gerd Behrmann\nTarget: master, 2.11, 2.10, 2.9, 2.8, 2.7, 2.6\nRequire-book: no\nRequire-notes: no","Tigran Mkrtchyan",true],["8c295eeec9da72895b1994d44eebff7b522edee7","6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","info: Reduce safety limit to 50 ms\n\nAfter reducing the DGA query separation to 100 ms, the DGA safety kick in and\nlimits the DGA frequency. This patch changes the safety limit to 50 ms (20 Hz).\n\nTarget: trunk\nRequest: 2.11\nRequest: 2.10\nRequire-notes: no\nRequire-book: no\nAcked-by: Paul Millar <paul.millar@desy.de>\nPatch: https://rb.dcache.org/r/7540/","Gerd Behrmann",true],["3dca7ee2823a8157214ca351aa2013822c909c85","d922616d71ca96f778c63ea49015b141afa0a2ae@desy.de","nfs: set file size in namespace on stable_how is FILE_SYNC4\n\nIn some workloads, nfs client may request file size before mover\nis slosed and file size is set. Nevertheless, prior such requests client\nmust commit written data by sending an explicit COMMIT operation\nor WRITE with FILE_SYNC flag.\n\nThis change covers both cases. To match pnfsid's to mover we keep\na dedicated mapping with an assumption, that only single mover can\nperform write operation.\n\n2.11 is requested as we see the need for a sync in production system at desy.\n\nAcked-by: Gerd Behrmann\nTarget: master, 2.11\nRequire-book: no\nRequire-notes: yes","Tigran Mkrtchyan",true],["b0f88c5080926e867ca644087a4e75e7c077bf00","6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","Upgrade to Milton 2.6\n\nAppears to be working. Solves the problem where proxied vector reads cause the entire\nfile to be written to local disk. Instead only the actual data that is to be returned\nto the client is buffered. If this is more than 100 KB, that data is buffered on local\ndisk. The file is deleted after successful return of the data.\n\nThere are still problems though: In case of failures reading the data from the pool,\nthe temporary file is not deleted. The entire file is still downloaded from the pool\nto the door (even when only the requested data is buffered), and the multi-range writer\nin Milton is extremely inefficient as it processes one byte at a time (it fails to\noverride the multi-byte write methods).\n\nThe new version supports using cookies to remeber when a client has already logged in.\nI disabled this feature for now.\n\nTarget: trunk\nRequest: 2.11\nRequest: 2.10\nRequest: 2.9\nRequest: 2.8\nRequest: 2.7\nRequest: 2.6\nRequire-notes: yes\nRequire-book: no\nAcked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>\nPatch: https://rb.dcache.org/r/7530/","Gerd Behrmann",true],["1d4fb5882fcaa898fe740ea983e9676a51d1b992","6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","ftp: Relax requirements for EPSV and EPRT\n\nSince using EPSV without delayed passive forces the use of a data channel\nproxy, we have rejected use of RFC 2428 commands over IPv4. This requirement\ncan however be relaxed: EPRT can be used over IPv4 without proxying, and\nEPSV can be used over IPv4 without proxying when delayed passive is enabled.\n\nThe patch has the potential to extend compatibility with clients and I hence\nrequest it to be backported to 2.10.\n\nTarget: trunk\nRequest: 2.11\nRequest: 2.10\nRequire-notes: yes\nRequire-book: yes\nAcked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>\nPatch: https://rb.dcache.org/r/7537/","Gerd Behrmann",true],["3f52dce613e1f76efe119da11a8fab18e711b2d8","6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","srm,webdav,httpd: Log SSL handshake failures\n\nWill log SSL failures, such as when clients try to use disabled features. Eg:\n\n28 nov. 2014 10:21:53 (SRM-Gerds-MacBook-Pro-5) [] SSL handshake with localhost/127.0.0.1:62713 failed: SSLv2Hello is disabled\n\nTarget: trunk\nRequest: 2.11\nRequire-notes: yes\nRequire-book: no\nAcked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>\nAcked-by: Paul Millar <paul.millar@desy.de>\nPatch: https://rb.dcache.org/r/7539/","Gerd Behrmann",true],["af625e3cf88fe40a1d009dc46c1c6018047511bd","6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","ftp: Relax requirements for EPSV and EPRT\n\nFix unit tests after 1d4fb58.","Gerd Behrmann",true],["42e31b5f935af4f3220aa29f2e51e8ea2c0bd146","6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","poolmanager: Fix full pool detection for WASS\n\nWhen a file cannot fit within the free plus removable space without\ncrossing the pool gap, a pool is considered full. WASS erroneously\nused the decayed removable space in this check. This meant that\nif all pools had very fresh files, they could all be considered\nfull even though there are plenty of removable files. This patch\nchanges the check to consider all removable space of a pool. The\ndecayed removable space is only used when weighting the random\nselection.\n\nTarget: trunk\nRequest: 2.11\nRequest: 2.10\nRequest: 2.9\nRequest: 2.8\nRequest: 2.7\nRequest: 2.6\nTicket: http://rt.dcache.org/Ticket/Display.html?id=8390\nRequire-notes: yes\nRequire-book: no\nAcked-by: Christian Bernardt <christian.bernardt@desy.de>\nPatch: https://rb.dcache.org/r/7531/","Gerd Behrmann",true],["952316ff36f15b7f4dec962ad279994dcd97bacd","d922616d71ca96f778c63ea49015b141afa0a2ae@desy.de","nfs: return correct verifier on WRITE and COMMIT\n\nAcked-by: Paul Millar\nAcked-by: Gerd Behrmann\nTarget: master\nRequire-book: no\nRequire-notes: no","Tigran Mkrtchyan",true]],"size":19,"ref":"refs/heads/master","head":"952316ff36f15b7f4dec962ad279994dcd97bacd"}