|  |
|--|
|  |  |  |  |  |
|--|--|--|--|--|
|                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |  |  |  |
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--|--|--|
| **Published by the DOI Departmental Consulting Archeologist/NPS Archeology Program, National Park Service, Washington, DC, September,           1993.**

*This Technical Brief describes objectives, organization,           and methods that could be used for conducting peer reviews in public           archeology projects. Each of these elements is illustrated with examples           drawn from peer reviews conducted by the Department of the Interior's           Departmental Consulting Archeologist between 1981 and 1990. The purpose           of this Technical Brief is to provide guidance to government agencies           and other archeological resources management programs on use of the           peer review process as one tool to improve the effectiveness of their           activities. It may be an especially important tool to help resolve conflicts           which may arise due to the need for interagency cooperation, the complexity           of the archeological work, or professional disagreements. This Technical           Brief demonstrates how peer reviews can address these typical problems           and explains ways the process can be implemented.*

**Introduction**

The review of research by peers has been one hallmark           of science throughout its history. Peer reviews are the conscience of           science; they provide a necessary check on its practitioners and an           explicit evaluation of their efforts. Other professions have recognized           the importance of peer reviews for applied sciences as well. For example,           the American Society of Civil Engineers and American Consulting Engineers           Council (ACEC) have developed a program that offers peer review services           for engineering projects and programs (ACEC 1990 a,b,c,d).

The Departmental Consulting Archeologist (DCA) and his           staff in the Archeological Assistance Program are authorized by the           Secretary of the Interior to provide leadership and coordination for           Federal and other types of public archeology. The Secretary is assigned           these roles by several statutes including the Antiquities           Act, Historic           Sites Act of 1935, National           Historic Preservation Act, and Archaeological           Resources Protection Act. The peer review of selected Federal archeological           projects is one among a number of DCA programs that provide leadership           and coordination (Canouts 1992; Canouts and McManamon 1991; Knoll 1991;           Knudson and McManamon 1992; McManamon, Smith, and Waldbauer 1990).

This Technical Brief describes the purposes and the           goals of the Federal archeology peer review process as conducted by           the DCA between 1981 and 1990. The government agencies and the professionals           who have been in these reviews recognize that peer reviews can provide           an important contribution to project design and management, thereby           advancing American archeology. This brief provides an example of the           peer review process in the Federal archeological arena, as opposed to           the academic domain. It also supplies information about the planning,           design, and implementation of peer reviews. Four           appendices with examples of documents illustrating efforts at planning           and implementing peer reviews are provided.

**Purposes           and Goals of the Federal Peer Review Process**

 In an academic research framework, as opposed to typical           public archeology projects, the principal investigator develops a research           program to investigate problems that are of theoretical, methodological,           and/or substantive interest. The researcher develops a research design,           identifies and obtains permission to investigate the sites that fit           the project's scientific needs, and raises the necessary funds by obtaining           a grant. The researcher plans and conducts the work and publishes the           results. Review of the scientific merits of the proposed work by his           or her peers occurs during the grant application process, in the review           of the results by the granting organization, as part of the publication           process, and in the reception of the product by the scientific community.           

On the other hand, archeology conducted to comply with           Federal historic and archeological preservation laws must follow sound           archeological practice as well as the regulations, procedures, and guidelines           mandated by and developed from those laws. Archeology conducted by these           authorities is to benefit the American people generally through the           preservation of important archeological resources and information. The           first objective of research in public archeology is to identify, evaluate,           and record significant archeological resources. The scientific information           is used for a variety of practical purposes, including education and           creating an appreciation of the nation's heritage among its citizens.           The concomitant benefits that individual archeologists may derive from           these endeavors with regard to professional standing and career advancement           are secondary.

Early in the historic preservation process, the Federal           agency identifies archeological properties within the boundaries of           an undertaking. This is accomplished through literature reviews, records           checks, and field surveys. The significance of the sites, especially           for the important archeological and historical data they contain, is           determined based upon eligibility for the National Register of Historic           Places. Significance commonly is assessed in reference to State comprehensive           historic preservation plans or historic contexts used by the State Historic           Preservation Officer (SHPO).

If important data will be lost because of the undertaking,           the Federal agency must consider this potential and consult with the           SHPO to develop ways to mitigate the loss. The agency also may have           to get the comments of the Advisory Council on Historic Preservation           (ACHP) before moving forward with the undertaking.

Several parameters have been defined for Federal archeological           projects that constrain the scientific freedom to develop research topics.           The principal difference between an exclusively academic research design           and one developed for a public project is that the former develops from           scientific curiosity while in the latter scientific research needs are           conditioned by the potential data content of the archeological site           being impacted by the public undertaking.

These constraints on the opportunity for an individual           researcher to develop theoretical, methodological, or substantive aspects           of the research design do not mean that Federal projects are devoid           of such concerns. On the contrary, the process of developing a research           design and a strategy to implement it, completing the work, and reporting           are the same general steps of any scientific research project. Also,           the development of the archeological project within Federal historic           preservation is a consultative process in the hands of Federal archeologists           and their counterparts in the SHPO office. Project plans and archeological           results also may be reviewed by archeologists and historic preservation           professionals of the ACHP. Frequently suggestions made during these           consultations result in design improvements. As a result of this coordination,           the project has a firm scientific foundation.
 **Figure 1.               Archeologists at site of Hohokam pioneer period pithouse.**              Photo courtesy Bureau of Reclamation/Tom Lincoln
Before the passage of the National Historic Preservation           Act of 1966 (NHPA), rescue or salvage archeology was handled without           input from the States or the ACHP. Archeological rescue work was usually           controlled by the time and money available rather than by the development           of thoughtful and explicit research designs. The National Park Service           (NPS), operating with its own professional staff through the River Basin           Survey program of the Smithsonian Institution or with academic contractors,           conducted the archeological salvage program of the nation. Private energy           developers and State highway departments contracted directly with academic           institutions to complete salvage archeology in their construction projects.           Competition among academic institutions for government, private utility           company, or highway department contracts for archeological work did           not exist during this period. Normally the NPS, the utility company,           or highway department approached an academic institution and asked the           institution to carry out whatever archeology was needed within the time           and funding available.

Today Federal agency archeologists or contractors from           the public and private sectors conduct Federal archeological projects.           Competition among archeological contractors is the norm unless the work           is done by agency personnel. It does not matter who does the project           for the Federal peer review process to work.

The perception has existed that Federal archeological           project research design is less open to evaluation and criticism than           an academic research project funded by a granting agency. This is hardly           the case. As mentioned, evaluations occur as part of memoranda of agreement,           National Register of Historic Places nominations, determinations of           effect, reviews of scopes of work for data recovery plans, and requests           for proposals that form the bases for contracts. Depending on the specifics           of the project, these evaluations are performed by archeologists, other           cultural resources specialists, and perhaps officials of various agencies.           These reviews represent a broad range of scientific and legal compliance           concerns. Frequently the evaluations improve the agency proposal to           mitigate the loss of important information, and stipulations for formal           peer reviews are often specifically included in memoranda of agreement           or agency contracts. This is all done before a project is announced           for bidding. As part of the proposal, the winning bidder also prepares           a research design or work plan. This provides refinements for conducting           the research. It also defines the protocols that will serve as project           guides. 

The primary goal of the peer review process developed           by the DCA is to evaluate the conduct of Federal archeological projects           and assess the competence and efficiency of projects relative to archeological           practice and legal compliance. An academic peer review rates a project           on its scientific merits alone; reviewers examine the theoretical basis           for the research, the data needed, and how they contribute to the research           goals. They evaluate the methodology used to get needed data and the           analytical methodology. They determine if the research was conducted           in an appropriate manner. In Federal archeology, the peer review examines           additional topics as well as these. Is the agency following the stipulations           of the compliance documents agreed to under the Section 106 process           of NHPA? Have data fulfilled the needs of the research design? Have           unexpected data or research domains come to light that need additional           investigation? Does the project need to be changed or redesigned to           take these findings into consideration? 

During the 1980s, Federal archeology project peer reviews           were undertaken when controversy stirred about an undertaking or the           archeology associated with it. Some reviews were conducted because of           major disagreement between the agency and the contractor about the work.           Allegations or questions may have come from the archeological profession,           the media, special interest groups, or members of Congress. More recently           agencies have requested reviews by the DCA to investigate whether projects           were adequate. One review was conducted for an agency that wished to           improve a project. 

In public archeology the use of the peer review process           also serves other purposes:
- Demonstrating to the professional community, Congress,             the Administration, and others that the Federal archeology program             produces excellent results;  Creating networks for communication between government             and academic archeologists that improve archeology;Helping resolve disputes between Federal agencies,             contractors, special interest groups, and the media;
- Providing Federal archeologists with professional             credibility among their managers and the academic community.      **Philosophy           of Peer Reviews for Public Projects**  In designing the peer review process the scientific           value of an archeological project, as determined by the NHPA Section           106 consultation, is accepted and used. Furthermore, the research goals           and priorities and the stated applications of archeological methodology           also are accepted and used. The peer review evaluates the project using           the following criteria:  - Is the archeological research consistent with the             needs of the research design?  Is it consistent with the contract, memorandum of             agreement, and other covering directives?Is the project on schedule?
- Is the project within budget?  Are the agency and the contractor fulfilling the             requirements of the contract?Are there unresolved disputes or disagreements between             the contractor and the agency related to the archeological work?
- Are the materials and data recovered consistent with             the needs of the research design?  Are there gaps in the data? If so, is it possible             to fill them?Is it desirable to suspend a particular line of research?
- Based upon new findings, should other lines of research             be pursued in place of or in addition to those identified in the research             design?  Are the public education and outreach aspects of             the project being done adequately and effectively?Do the steps being taken to clean, catalog, describe,             and analyze excavated material conform to the plans for ultimate,             long-term curation of the material and data?
This approach accepts previous decisions made by the           various consulting parties to mitigate the loss of important archeological           information caused by a Federal undertaking. It also provides an occasion           to assess the project to determine if any changes are desirable. This           approach steers clear of second-guessing earlier decisions or trying           to identify all of the approaches for conducting archeological investigations.           The manager of the peer review and the agency archeologist establish           a project review agenda and develop a list of suggested review topics.           These aid the peer review panel in examining archeology defined by the           NHPA Section 106 process.
**Project           Selection**
Since 1981, the DCA has conducted six peer reviews (Table           1). The experience from conducting these reviews is the basis for this           Technical Brief. These projects were selected for a variety of reasons.           The Dolores Reservoir Project was reviewed because of problems between           the contractor and the agency about the scope of the investigation and           project management. The Central Arizona Project was selected because           the Bureau of Reclamation wanted to confirm that the scope and focus           of the archeological project were appropriate. The Jackson Lake Dam           Project was evaluated to demonstrate to the members of the Wyoming Congressional           delegation that the work was adequate and satisfied State interests.           The Fish and Wildlife Service sought a peer review to address Native           American concerns and to assure that historic preservation requirements           were being met in the Stillwater Wildlife Management Area Project. The           U.S. Army Corps of Engineers wanted to confirm that its historic preservation           responsibilities were being fulfilled in a manner sensitive to Native           American religious concerns in the Libby Dam Project. The Soil Conservation           Service wanted an evaluation of the Alkali Creek Project in light of           SHPO and professional community concerns.
**Table               1. Projects for Which Peer Reviews Were Done by the Departmental               Consulting Archeologist, Archeological Assistance Division, National               Park Service***Project/Location**Year* *Agency* Dolores Project, McPhee Reservoir, CO                 1981    Bureau of Reclamation   Central Arizona Project, AZ 1986Bureau of ReclamationJackson Lake Dam, Grand Teton National Park, WY               1987National Park ServiceStillwater Wildlife Management Area, Carson Desert,               NV1988Fish and Wildlife ServiceLibby Dam Project, Lake Koocanusa, MT 1989Corps of EngineersAlkali Creek Project, ND1990Soil Conservation Service
**Organizing           and Conducting Peer Reviews**
A peer review of a project is an activity that must           be developed harmoniously as a partnership between the peer review manager           and the agency. For all but the Alkali Creek peer review, the DCA served           as peer review manager. In the case of Alkali Creek, a senior NPS archeologist           well acquainted with the project area was designated by the DCA as peer           review manager. Normally the agency representative has been the archeologist           responsible for the administration and conduct of the agency's archeological           program, although another agency official may serve in this role. The           need for frequent and clear communication between the peer review manager           and the agency archeologist or other official in charge of the project           is mandatory. Good communication is an absolute necessity for a successful           peer review. No matter who serves in them, these roles are essential           components of the peer review process.
All aspects of the peer review are developed through           close coordination between the peer review manager and the agency. Both           parties must agree on the scope of the review. Frequent and open communication           is necessary in all aspects of the project. The agency archeologist           needs information about the progress of the planning effort by the peer           review manager to develop agency plans. The peer review manager also           discusses the peer review with the SHPO and ACHP to gather information           that should be supplied to the team. Frequent communication bolsters           the commitment made by both parties to assure that the necessary actions           are completed in a timely manner. In undertaking a peer review, one           of the peer review manager's goals is to create a partnership with the           agency. The creation of such a partnership is the principal ingredient           that has made the process, as conducted by the DCA, successful.
Once a project is selected, the following tasks, at           a minimum, must be undertaken to conduct the peer review:
- Defining the review's scope and schedule.  Determining the review's cost and securing funds.Planning the team's composition and choosing members.
- Compiling documentation and mailing to the team.  Scheduling the on-site visit.- Arranging travel and lodging.  Reserving meeting space.Making appointments for interviews.
- Preparing the agenda and itinerary.  Identifying review topics.Developing support for the team.
- Making arrangements for office space.  Securing equipment for team use.  Conducting the on-site visit.Preparing the draft final report (by peer review             manager).
Obtaining revisions to the final draft report                 from the review team.
- Revising manuscript.  Obtaining review team approval of final report.Producing the final report.
- Compiling recommendations for the agency. Completing                 remaining administrative tasks.   Providing final report to agency officials.
The following sections of this Technical Brief will           provide information about each of these tasks. The examples cited are           drawn from peer reviews conducted in the past by the DCA.

**1. Defining           the Scope of the Peer Review**          The reasons for selecting a project for peer review normally defines           the scope of the review. If charges of inadequacy have been made, the           peer review will focus on the conduct of the archeological investigations           under the agreements and documents covering the project. If the peer           review is a response to problems between the agency and the contractor,           the peer review will focus on these matters. The topics identified for           review for the Stillwater Wildlife Management Area are provided as examples           in Appendix A; those for the Jackson           Lake Dam project were similar. In any case, the scope of the peer review           should focus on identifying solutions to problems and not foster inappropriate           or adversarial investigative roles.

**2. Cost           of the Peer Review and Funding**          Arrangements must be made to fund the various costs for peer reviews           and establish the budget parameters within which the peer review manager           must operate. The costs of peer reviews conducted by the DCA, for instance,           have been covered by various means, including the DCA's annual operating           budget, shared expenses of the DCA and the agency, and full funding           by the agency. When an agency shares or covers the costs, the funds           typically come from the project budget. Cost sharing between the DCA           and the agency has been the most common arrangement. 

Honoraria or consulting fees have not been paid to members           of the DCA peer review panels. Payment of honoraria to non-Federal members           of the team is legal, but it is illegal to pay a Federal team member.           It seemed inappropriate to compensate some team members but not others.           Furthermore, there was concern that if honoraria were paid the peer           review team might not feel that it could be critical and candid in its           assessment. There was also a potential for the review to appear prejudiced.

The costs of transportation and per diem of the peer           review members must be considered. The purchase of airline tickets may           take a large portion of the funds available. Other costs include: reproduction           and postage to provide project information to the peer review team;           on-site equipment, materials, and work space for the team; administrative           support; and final report preparation and distribution. Many of these           costs can be covered in-kind by the agency or the peer review manager.

**3. Composition           and Selection of the Peer Review Team**          The first peer review team assembled by the DCA consisted of three nationally           recognized archeologists from academic positions. Two were experts in           the archeology of the project region. One was an expert from outside           the area. This composition was modified for later peer reviews. It was           learned that academic archeologists were unfamiliar with the NHPA Section           106 process and unaware of many of the legal requirements of Federal           historic preservation. They had little or no knowledge of Federal procurement           and contracting regulations and, therefore, they found it difficult           to readily identify efficient, effective solutions to problems related           to these topics. 

For later reviews, an archeologist from a public agency,           but one different from the one under review, was added to the team.           This served the additional goal of increasing the interaction among           government archeologists and their academic colleagues. This was a way           to improve the credibility of the government archeologists with their           academic counterparts. We have found that a balanced peer review team           consists of one public archeologist from an agency other than the one           responsible for the project, an academic expert in the archeology of           the project area or region, and an academic expert from outside the           research area. 

To identify potential review team members the peer review           manager and the agency archeologist independently compile three lists           of candidates for: (1) the regional academic expert, (2) the outside           academic expert, and (3) the public agency archeologist. They consolidate           their lists and decide which candidates and alternates to invite. 

Care is to be given in selecting members of the panel.           Beyond a candidate's scientific credentials and availability, other           elements are considered, for example: 

- Does the candidate have a reputation for completing             work in a reasonable time?   Does the candidate have a previous relationship or             any other consideration with the agency or the project that could             be construed as a conflict of interest? Was the candidate an unsuccessful bidder for the             contract? 
- Does the candidate work well with others?   While these elements may seem straightforward, ignoring             them could make completing an acceptable peer review difficult or             affect its timeliness.
Once the peer review manager and agency archeologist           agree on the candidates and the approximate dates for the review, the           peer review manager telephones the candidates to determine their willingness           and availability. Acceptance of the invitation is followed by a formal           letter from the peer review manager providing information about the           project, the schedule, and other details.

**4. Compiling           Documentation for the Peer Review Team**          By the time of the onsite visit the peer review team should have a thorough           knowledge of the project. The agency responsible for the project compiles           and furnishes the panel with the documents that describe the undertaking,           the archeological resources, their significance, the research questions,           and background information about the local archeology. Acquisition of           this necessary knowledge by the peer review team is facilitated by providing           all the documentation to team members at least three weeks before the           on-site visit. If it is provided with less time before the on-site visit           the peer review team will not have time to review this basic information.

The package of material compiled for the orientation           of the peer review team will vary from project to project. Typically           it will consist of survey and evaluation reports and NHPA Section 106           documentation, such as the Memorandum of Agreement or Determination           of No Adverse Impact, scope of work or request for proposal, and research           design. Interim or annual progress project reports, if available, also           are provided. Additionally, correspondence and internal documents may           be included. (A list of the kinds of materials supplied for the Central           Arizona Project and Stillwater National Wildlife Management Area peer           reviews is provided in Appendix           B.)
 **Figure 2.               Archeologists sharing information at site of Hohokam village.**              Photo courtesy Bureau of Reclamation/              Tom Lincoln
**5. Scheduling           the Peer Review**The ideal time for a peer review is at the midpoint           of a project. Typically, field and laboratory work has proceeded to           the stage where the review team can make a reasonable evaluation of           the results. A peer review at the beginning of a project is premature           because there will be few results to assess. If the project is near           completion the important recommendations of the peer review team will           be difficult to implement; usually time and money have become short           at this stage.

A second consideration in scheduling the peer review           is for a time that is acceptable to potential team members. It is useful           to identify a 3- to 4-week period for the peer review approximately           3 months in advance of the on-site visit. Within such a time frame it           is usually possible to fit the schedules of all personnel involved.           Situations that can affect the scheduling of the review must be considered.           These include college and university calendars or special events like           conventions or athletic contests in the project area. These situations           can affect travel arrangements, as lodging may be difficult to find           in such circumstances. The schedules and commitments of the individuals           representing the various organizations who will be interviewed by the           peer review team must be considered. The on-site visit must be scheduled           so that those who will travel from outside the project area, the SHPO           or appropriate staff, ACHP personnel, and other interested parties,           can participate.

The peer review manager, in consultation with the agency           archeologist, develops an agenda for the peer review team's on-site           visit. A detailed agenda assures that the on-site visit is carefully           structured. Each interview focuses on a specific task or topic. Each           interview session is allotted a specific amount of time. Sessions usually           do not exceed two hours without a break. Maintaining strict adherence           to the agenda assures that the peer review team stays focused on the           task and adheres to the schedule.

During the first session of the on-site visit the peer           review manager, the agency archeologist, and the peer review team discuss           the agenda in detail and modify it as necessary. Implementing modifications           in the agenda may require complex changes in the interview schedule.           It is important to determine the final agenda as early as possible.

**6. On-Site           Visit**          The peer review manager and the agency archeologist plan the on-site           visit to use everyone's time efficiently. Usually the peer review manager,           agency personnel, and peer review team arrive the day before the on-site           visit begins.

The peer review manager and the agency archeologist           provide the necessary administrative support to the peer review team.           This assures that an efficient review occurs. Meeting facilities, telephones,           transportation, and lodging and dining facilities must be available.           Secretarial assistance, equipment, and supplies may be needed. The object           of taking care of these necessities is to allow the team to devote its           energies to the peer review. Personal computers have been very useful.           In providing these kinds of administrative support, the peer review           manager and agency function as aides to the team.

The first part of the agenda is devoted to briefings           by the peer review manager and agency personnel. The peer review manager           describes and then emphasizes the specific scope of work of the peer           review (see Appendix C for           examples). The peer review team can examine any aspects of the project           related to the archeological work. The plan for the rest of the on-site           visit will depend on the schedules of individuals, the weather, field           conditions, and other variables. During the week-long peer reviews conducted           by the DCA, personnel interviews and field visits normally were concluded           by the end of the third day so the team could review its findings and           draft its report. Typically the draft report was completed by the evening           of the fourth day. Exit interviews with the DCA, agency personnel, and           other interested parties were conducted on the fifth day.

The peer review team normally interviews agency personnel           such as the project engineer, environmental staff, and the archeologist.           Other historic preservation specialists, the contracting officer, or           contracting officer's technical representative may be interviewed. The           team meets with the contractor or principal investigator and the principal           investigator's senior staff, such as field directors, crew chiefs, data           control manager, and laboratory director. The team also meets with the           interdisciplinary consultants, such as geologists, geomorphologists,           pedologists, paleobotanists, paleozoologists, and physical anthropologists.           The team usually will visit the site or sites currently under investigation           and the laboratory facilities of the contractor to get a feel for the           day-to-day operation and management of the project. In some instances           the peer reviewers will meet with some members of the professional community           and Indian Tribes or their representatives. The team also may interview           interested members of the public and Congressional staffers.
 **Figure 3.               Re-excavation conference with contractor and peer review team, Central               Arizona project.**              Photo courtesy Bureau of Reclamation/              Dan McKeever
Each peer review will vary regarding the parties interviewed           because the focus of the peer review will be different in each project.           In the peer reviews conducted by the DCA, it was standard procedure           to provide the SHPO and ACHP with a chance to meet with the peer review           team. Sometimes this was done by telephone interviews. This consideration           is based on the regulatory roles of these agencies in historic preservation.           Furthermore, the professional staff of the SHPO are especially knowledgeable           about their State's archeological resources and can provide important           perspectives to the team.

The scope of the peer review and the specific circumstances           will determine whether or not the agency archeologist or official attends           all of the interviews conducted by the team. In cases where problems           exist between the agency and the contractor, SHPO, ACHP, or other parties,           it has been found prudent for the team to meet both with and without           the agency personnel. The absence of agency personnel may allow for           open and candid discussion that otherwise would be difficult. 

The on-site visit is a period of intense work by the           peer review team. Every effort is made to keep the team on schedule           and focused on its task. The peer review manager and agency personnel           provide all reasonable support requested to assure that the team's tasks           are completed by the scheduled time.

**7. Peer           Review Draft Report**          The peer review manager and the agency archeologist prepare a proposed           outline for the report, making it clear to the peer review team that           the topics and proposed outline are only suggestions. The content of           the final report is determined by the peer review team. The purpose           of the proposed report outline is to provide the team with guidance.           The interviews provide specific and important information and views           from a variety of perspectives. The peer review manager always assures           the team in each review that the content and recommendations of the           final report are entirely their responsibility. The team is welcome           to change or reorganize the outline and format. Team members may add           or drop topics as they think necessary or appropriate, though departures           from the model usually concern format or organization.

The importance of coordination and careful attention           to details in production of the final draft report cannot be overestimated           by the peer review manager. Some cautionary examples are cited from           the peer reviews conducted by the DCA to demonstrate the importance           of organized, considered activities and then continuing to learn from           experience.

The peer review team for the Dolores Project at McPhee           Reservoir spent an intensive week interviewing various parties concerned           with the project. After the on-site visit the team members returned           to their regular employment and prepared the draft final report. As           a long-distance effort, completion of the draft was delayed considerably.           This delay was caused by factors such as lack of communication between           the members of the team, the time taken to circulate drafts and comments           sequentially through the mail, and other commitments or priorities of           the panel members. The Dolores Project peer review team was not provided           with a proposed report outline, so team members had to develop an outline           and report format, which proved to be very time-consuming. Based on           this experience the DCA determined it would be helpful in the future           to provide guidance and suggested topics to cover. This practice has           expedited the preparation of draft peer review reports.

The Central Arizona Project peer review team decided           that it would complete a draft report by the end of the on-site visit.           Despite certain misgivings, this proposal was encouraged and the draft           report was delivered as promised. The DCA edited the draft report for           grammar and worked with the format, but made no changes in content.           A revised draft was mailed simultaneously to the authors for revision.           Marked and revised copy was returned to the DCA and changes made. The           final report was produced and distributed to the Bureau of Reclamation           in a short period along with the DCA's own recommendations. Other interested           parties were sent copies as part of this distribution. The Central Arizona           Project peer review demonstrated the clear value of requiring the team           to complete a draft report by the end of the on-site visit.

Appendix D           provides examples of report outlines presented to the peer review teams           for the Central Arizona and Carson Desert projects.

**8. The           Final Report**          The production of the final report of the peer review can be a rather           simple, but intense task. The draft report is edited for style and format           and unclear language is revised. All changes in the manuscript are made           using the "strikeout," "bold" or other options of the chosen word processing           program. This allows the author to identify easily the changes that           have been made in the revised text. This draft is mailed simultaneously           to the authors for their corrections and approval. The peer review manager           makes the changes required by the authors and produces the final report.           Depending on the quality of the draft report, production of the final           version requires some 16 to 24 person hours of effort.

The peer review manager transmits the report and recommendations           for action to the agency for consideration. The peer review manager           may wish to include his or her own recommendations, which address the           findings and recommendations of the peer review team and may be mandated           according to the authorities by which the peer review manager undertook           this task. While the peer review manager normally endorses most of the           recommendations made by the peer review team, occasionally it will be           determined that the recommendations do not really improve the project           when compared to costs by the agency to implement them. Irrespective           of whether the peer review manager supports or rejects the recommendations           of the team, a clear explanation is always provided. 

**Conclusion**

Peer reviews for public archeology projects have been           and will continue to be important tools for agencies to use to improve           both research and preservation activities. They assist agencies in meeting           the objectives for which public archeology projects are undertaken.           In addition, they can help improve public awareness of the value of           contributions to knowledge about the nation's cultural past that are           central to such projects. When peer reviews are conducted in this comprehensive           way they have had two important long-term results. First, the agency           gains a better perspective on the impacts of sound archeological resources           management and therefore can develop effective means to improve its           programmatic efforts. Second, the benefits of public archeology research           are clarified for the academic disciplines and therefore can be incorporated           appropriately into the most current theoretical and interpretive developments           for explaining the archeological record.

Implicit also in the purpose of this Technical Brief           to provide guidance to governmental agencies and other archeological           resources management programs on the peer review process is encouragement           for those agencies to undertake their own. Such peer reviews may be           full-scale versions, similar to ones completed by the DCA, or they may           be smaller, to fit the needs of more localized projects. In either case,           the results can lead to cost savings and increased support for archeology           among members of the public and within agencies.

*Return to*Technical           Briefs | Text | Reports and References           | Appendices

**introduction**

**purposes                 and goals of the federal peer review process**

**philosophy                 of peer reviews for public projects**

**project                 selection**

**organizing                 and conducting peer reviews**

**conclusion**

**dca                 peer review reports to 1993**

**references                 cited**

**appendices**

**print                 version  **

**technical briefs                 index**
DOI |			History & Culture | 			Search | 			Contact | 			FOIA |			Privacy |			Disclaimer |			USA.gov
 MJB/EJL 
-  archeology       in the parks   
-  archeology       nationwide   
-  professional       tools   
-  for       the public   
-  kennewick       man 
-  the       antiquities act 
-  what's       new 
-             
-             
-  technical assistance 
-  databases 
-  standards & guidelines 
-  training & education 
-  reports 
-  publications 
-             
-             
-             
-  features 
-  learn and participate 
-  for teachers |