daitss/core — Issue #639: storage: investigate storage problem (EKAG48Q5K_093DG9)
Author: lydiam
Created: 2012-03-02 20:41:29+00:00
State: open

Package EKAG48Q5K_093DG9 is over 200GB in size and has failed DAITSS storage several times. We made every effort to provide sufficient /tmp and other disk space to process the package, but in the most recent attempt there appears to have been a problem with space in darchive:/var/daitss/tmp, whereas the prior two attempts failed due to a timeout.

The timeline for EKAG48Q5K_093DG9 from core.log (processing was restarted via the GUI):

2012 Mar 1 09:11:36 fclnx30 Core[32463]: INFO fda.fcla.edu: Parsing AIP descriptor for EKAG48Q5K_093DG9
2012 Mar 1 09:15:09 fclnx30 Core[32463]: INFO fda.fcla.edu: Reserving storage for EKAG48Q5K_093DG9
2012 Mar 1 09:15:09 fclnx30 Core[32463]: INFO fda.fcla.edu: Writing AIP for EKAG48Q5K_093DG9 to storage
2012 Mar 2 08:52:35 fclnx30 Core[19263]: INFO fda.fcla.edu: Parsing AIP descriptor for EKAG48Q5K_093DG9
2012 Mar 2 08:56:30 fclnx30 Core[19263]: INFO fda.fcla.edu: Reserving storage for EKAG48Q5K_093DG9
2012 Mar 2 08:56:30 fclnx30 Core[19263]: INFO fda.fcla.edu: Writing AIP for EKAG48Q5K_093DG9 to storage

darchive storage.log entries:

[lydiam@fclnx30 web-services]$ grep EKAG48Q5K_093DG9 storage.log
2012 Mar 1 09:13:18 fclnx30 StorageMaster[18567]: INFO storage-master.fda.fcla.edu: Rack: 128.227.228.197 - - [01/Mar/2012 09:13:18] "GET /search?pattern=EKAG48Q5K_093DG9 " 200 615 0.1741
2012 Mar 1 09:45:36 fclnx30 StorageMaster[18567]: INFO storage-master.fda.fcla.edu: Rack: 128.227.228.197 - - [01/Mar/2012 09:45:36] "GET /search?pattern=EKAG48Q5K_093DG9 " 200 615 0.1859
2012 Mar 1 11:39:42 fclnx30 StorageMaster[18575]: INFO storage-master.fda.fcla.edu: Request Received: 233951436800 128.227.228.113 - - "PUT /packages/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 16:07:31 fclnx30 StorageMaster[18567]: INFO storage-master.fda.fcla.edu: Rack: 128.227.228.184 - - [01/Mar/2012 16:07:31] "GET /search?pattern=EKAG48Q5K_093DG9 " 200 615 2.7716
2012 Mar 1 21:53:46 fclnx30 SiloPool[16916]: INFO silos.darchive.fcla.edu: Request Received: 233951436800 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: Internal Server Error - Store::SiloError # can't write the 'data' metafile for EKAG48Q5K_093DG9.006: No space left on device - /var/daitss/tmp/RackRewindableInput20120301-16916-2us1zf-0 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/releases/20111025150218/lib/store/silo.rb:149:in `put' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/releases/20111025150218/lib/store/silo.rb:384:in `write_lock' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/releases/20111025150218/lib/store/silo.rb:382:in `open' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/releases/20111025150218/lib/store/silo.rb:382:in `write_lock' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/releases/20111025150218/lib/store/silo.rb:97:in `put' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/releases/20111025150218/lib/store/silodb.rb:58:in `put' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: ./lib/app/posts.rb:18:in `POST /create/:name' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/releases/20111025150218/lib/store/poolreservation.rb:39:in `initialize' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: ./lib/app/posts.rb:17:in `new' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: ./lib/app/posts.rb:17:in `POST /create/:name' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:1211:in `call' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:1211:in `compile!' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:772:in `[]' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:772:in `route!' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:788:in `route_eval' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:772:in `route!' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:821:in `process_route' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:819:in `catch' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:819:in `process_route' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:771:in `route!' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:770:in `each' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:770:in `route!' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:886:in `dispatch!' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:706:in `call!' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:871:in `invoke' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:871:in `catch' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:871:in `invoke' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:706:in `call!' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:692:in `call' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/rack-1.3.5/lib/rack/commonlogger.rb:20:in `call' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/rack-protection-1.1.4/lib/rack/protection/xss_header.rb:22:in `call' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/rack-protection-1.1.4/lib/rack/protection/path_traversal.rb:16:in `call' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/rack-protection-1.1.4/lib/rack/protection/json_csrf.rb:17:in `call' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/rack-protection-1.1.4/lib/rack/protection/base.rb:47:in `call' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/rack-protection-1.1.4/lib/rack/protection/xss_header.rb:22:in `call' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/rack-1.3.5/lib/rack/nulllogger.rb:9:in `call' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/rack-1.3.5/lib/rack/head.rb:9:in `call' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/rack-1.3.5/lib/rack/methodoverride.rb:24:in `call' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:1334:in `call' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:1403:in `synchronize' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/sinatra-1.3.1/lib/sinatra/base.rb:1334:in `call' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/ruby/gem-repos/gems/passenger-3.0.7/lib/phusion_passenger/rack/request_handler.rb:96:in `process_request' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/ruby/gem-repos/gems/passenger-3.0.7/lib/phusion_passenger/abstract_request_handler.rb:513:in `accept_and_process_next_request' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/ruby/gem-repos/gems/passenger-3.0.7/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/ruby/gem-repos/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_spawner.rb:205:in `start_request_handler' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/ruby/gem-repos/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_spawner.rb:78:in `spawn_application' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/ruby/gem-repos/gems/passenger-3.0.7/lib/phusion_passenger/utils.rb:479:in `safe_fork' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/ruby/gem-repos/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_spawner.rb:63:in `spawn_application' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/ruby/gem-repos/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:264:in `spawn_rack_application' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/ruby/gem-repos/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/ruby/gem-repos/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/ruby/gem-repos/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:357:in `__send__' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/ruby/gem-repos/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/ruby/gem-repos/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously' 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: ERROR silos.darchive.fcla.edu: /opt/ruby/gem-repos/gems/passenger-3.0.7/helper-scripts/passenger-spawn-server:99 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 22:42:00 fclnx30 SiloPool[16916]: INFO silos.darchive.fcla.edu: Rack: 128.227.228.113 - - [01/Mar/2012 22:42:00] "POST /create/EKAG48Q5K_093DG9.006 " 500 62 2894.0213
2012 Mar 1 23:00:40 fclnx30 StorageMaster[18575]: ERROR storage-master.fda.fcla.edu: 500 Internal Service Error - Store of package EKAG48Q5K_093DG9.006 to http://silos.darchive.fcla.edu:70/create/EKAG48Q5K_093DG9.006 failed with status 500 - Internal Server Error; response from server was: #012

#012

500 Internal Server Error

#012

#012

Internal Server Error

#012

The server encountered an internal error or#012misconfiguration and was unable to complete#012your request.

#012

Please contact the server administrator,#012 root@localhost and inform them of the time the error occurred,#012and anything you might have done that may have#012caused the error.

#012

More information about this error may be available#012in the server error log.

#012

#012

Apache Server at silos.darchive.fcla.edu Port 70

#012

. 128.227.228.113 - - "PUT /packages/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 23:00:40 fclnx30 StorageMaster[18575]: INFO storage-master.fda.fcla.edu: Rack: 128.227.228.113 - - [01/Mar/2012 23:00:40] "PUT /packages/EKAG48Q5K_093DG9.006 " 500 814 40858.1156

XYMON alerts from that time period:

Thu Mar 1 22:10:09 EST 2012 - Filesystems NOT ok
yellow /var/daitss/tmp (81% used) has reached the WARNING level (80%)
Filesystem 1024-blocks Used Available Capacity Mounted on
/dev/mapper/daitss--vg15-var--daitss--tmp--lv1 727634640 576015736 136840960 81% /var/daitss/tmp

Next alert:
Thu Mar 1 22:25:17 EST 2012 - Filesystems NOT ok
red /var/daitss/tmp (90% used) has reached the PANIC level (90%)
Filesystem 1024-blocks Used Available Capacity Mounted on
/dev/mapper/daitss--vg15-var--daitss--tmp--lv1 727634640 641185128 71671568 90% /var/daitss/tmp

Next alert:
Thu Mar 1 22:45:26 EST 2012 - Filesystems NOT ok
yellow /var/daitss/tmp (84% used) has reached the WARNING level (80%)
Filesystem 1024-blocks Used Available Capacity Mounted on
/dev/mapper/daitss--vg15-var--daitss--tmp--lv1 727634640 592267532 120589164 84% /var/daitss/tmp

In addition, tarchive's silo service appears to have gone down as well:
Thu Mar 1 23:00:39 EST 2012
Tarchive Silo service is DOWN

From tarchive's storage.log:
2012 Mar 1 15:25:09 fclnx31 SiloPool[8910]: INFO silos.tarchive.fcla.edu: Request Received: 233951436800 128.227.228.113 - - "POST /create/EKAG48Q5K_093DG9.006 HTTP/1.1"
2012 Mar 1 16:13:31 fclnx31 SiloPool[8914]: INFO silos.tarchive.fcla.edu: Rack: 128.227.228.184 - - [01/Mar/2012 16:13:31] "GET /071/data/?search=EKAG48Q5K_093DG9&page=1 " 200 850 0.7016
2012 Mar 1 16:57:51 fclnx31 SiloPool[8922]: INFO silos.tarchive.fcla.edu: Rack: 128.227.228.184 - - [01/Mar/2012 16:57:51] "GET /071/data/?search=EKAG48Q5K_093DG9&page=1 " 200 850 0.0709
2012 Mar 1 19:35:26 fclnx31 SiloPool[8910]: INFO silos.tarchive.fcla.edu: Rack: 128.227.228.113 - - [01/Mar/2012 19:35:26] "POST /create/EKAG48Q5K_093DG9.006 " 201 358 15016.5252
2012 Mar 1 22:59:52 fclnx31 SiloPool[8914]: INFO silos.tarchive.fcla.edu: Using hostname 'silos.tarchive.fcla.edu' for silo lookup: 72 silos found.
2012 Mar 1 22:59:52 fclnx31 SiloPool[8914]: INFO silos.tarchive.fcla.edu: Using hostname 'silos.tarchive.fcla.edu' for silo lookup: 72 silos found.
2012 Mar 1 22:59:52 fclnx31 SiloPool[8914]: INFO silos.tarchive.fcla.edu: Rack: 128.227.228.113 - - [01/Mar/2012 22:59:52] "GET /services " 200 20962 0.6988
2012 Mar 1 23:00:02 fclnx31 SiloPool[8914]: INFO silos.tarchive.fcla.edu: Request Received: 164372480 128.227.228.113 - - "POST /create/E20090725_AAACXG.000 HTTP/1.1"
2012 Mar 1 23:00:40 fclnx31 SiloPool[8910]: INFO silos.tarchive.fcla.edu: Rack: 128.227.228.113 - - [01/Mar/2012 23:00:40] "DELETE /071/data/EKAG48Q5K_093DG9.006 " 204 - 363.9985
2012 Mar 1 23:00:48 fclnx31 SiloPool[8792]: INFO silos.tarchive.fcla.edu: Rack: 128.227.228.113 - - [01/Mar/2012 23:00:48] "POST /create/E20090725_AAACWW.000 " 201 355 153.5819

The DAITSS error message when this package failed:
ingest error	Thu Mar 01 2012 11:00:40 PM
bad status
http://storage-master.fda.fcla.edu:70/packages/EKAG48Q5K_093DG9.006: 500
500 Internal Service Error - Store of package EKAG48Q5K_093DG9.006 to http://silos.darchive.fcla.edu:70/create/EKAG48Q5K_093DG9.006 failed with status 500 - Internal Server Error; response from server was:

500 Internal Server Error

Internal Server Error

The server encountered an internal error or
misconfiguration and was unable to complete
your request.

Please contact the server administrator,
root@localhost and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.

More information about this error may be available
in the server error log.

Apache Server at silos.darchive.fcla.edu Port 70

.

It's not clear to me whether the /var/daitss/tmp problem or the perceived tarchive silo server downtime caused the SNAFU.

Note the second PUT attempt from darchive's storage.log that came after the long string of other errors:
2012 Mar 1 23:00:40 fclnx30 StorageMaster[18575]: INFO storage-master.fda.fcla.edu: Rack: 128.227.228.113 - - [01/Mar/2012 23:00:40] "PUT /packages/EKAG48Q5K_093DG9.006 " 500 814 40858.1156