Probabilistic Photometric Redshifts In The Era Of Petascale Astronomy
Creators: name: Carrasco Kind, Matias
Publisher: name: Zenodo
Year: 2015
Subjects: Photometric redshift
machine learning
data mining
galaxy surveys
petascale astronomy

description: With the growth of large photometric surveys, accurately estimating photometric redshifts, preferably as a probability density function (PDF), and fully understanding the implicit systematic uncertainties in this process has become increasingly important. These surveys are expected to obtain images of billions of distinct galaxies. As a result, storing and analyzing all of these photometric redshift PDFs will be non-trivial, and this challenge becomes even more severe if a survey plans to compute and store multiple different PDFs. In this thesis, we have developed an end-to-end framework that will compute accurate and robust photometric redshift PDFs for massive data sets by using two new, state-of-the-art machine learning techniques that are based on a random forest and a random atlas, respectively. By using data from several photometric surveys, we demonstrate the applicability of these new techniques, and we demonstrate that our new approach is among the best techniques currently available. We also show how different techniques can be combined by using novel Bayesian techniques to improve the photometric redshift precision to unprecedented levels while also presenting new approaches to better identify outliers. In addition, our framework provides supplementary information regarding the data being analyzed, including unbiased estimates of the accuracy of the technique without resorting to a validation data set, identification of poor photometric redshift areas within the parameter space occupied by the spectroscopic training data, and a quantification of the relative importance of the variables used during the estimation process. Furthermore, we present a new approach to represent and store photometric redshift PDFs by using a sparse representation with outstanding compression and reconstruction capabilities. We also demonstrate how this framework can also be directly incorporated into cosmological analyses. The new techniques presented in this thesis are crucial to enable the development of precision cosmology in the era of petascale astronomical surveys.
description: {"references": ["Abazajian K. N. et al., 2009, ApJS, 182, 543 (Cited on page 10.) Abdalla F. B., Banerji M., Lahav O., Rashkov V., 2011, MNRAS, 417, 1891 (Cited on page 3.) Abrahamse A., Knox L., Schmidt S., Thorman P., Tyson J. A., Zhan H., 2011, ApJ, 734, 36 (Cited on pages 6 and 138.) Abramowitz M., Stegun I., 1972, Handbook of Mathematical Functions: With Formulas, Graphs, and Mathematical Tables, Applied mathematics series. Dover Publications (Cited on page 126.) Adelman-McCarthy J. K. et al., 2008, ApJS, 175, 297 (Cited on page 24.) Ahn C. P. et al., 2013, ArXiv e-prints (Cited on pages 11 and 14.) Aihara H. et al., 2011, ApJS, 193, 29 (Cited on page 2.) Arnouts S. et al., 2002, MNRAS, 329, 355 (Cited on pages 3 and 149.) Assef R. J. et al., 2010, ApJ, 713, 970 (Cited on page 3.) Ball N. M., Brunner R. J., Myers A. D., Strand N. E., Alberts S. L., Tcheng D., 2008, ApJ, 683, 12 (Cited on page 4.) Ball N. M., Brunner R. J., Myers A. D., Strand N. E., Alberts S. L., Tcheng D., Llor`a X., 2007, ApJ, 663, 774 (Cited on page 4.) Banerji M. et al., 2014, ArXiv e-prints (Cited on page 15.) Baum W. A., 1962, in IAU Symposium, Vol. 15, Problems of Extra-Galactic Research, McVittie G. C., ed., p. 390 (Cited on pages 3 and 74.) Bender R. et al., 2001, in Deep Fields, Cristiani S., Renzini A., Williams R. E., eds., p. 96 (Cited on page 149.) Ben\u00b4\u0131tez N., 2000, ApJ, 536, 571 (Cited on pages 3, 74, 75, 79, 80, 85, and 149.) Bernstein G., Jain B., 2004, ApJ, 600, 17 (Cited on page 2.) Blake C., Collister A., Bridle S., Lahav O., 2007, MNRAS, 374, 1527 (Cited on pages 138, 158, and 159.) Blake C. et al., 2011, MNRAS, 418, 1707 (Cited on page 2.) Bolzonella M., Miralles J.-M., Pell\u00b4o R., 2000, A&amp;A, 363, 476 (Cited on page 3.) Bond J. R., Jaffe A. H., Knox L., 1998, PhRvD, 57, 2117 (Cited on page 158.) Bonfield D. G., Sun Y., Davey N., Jarvis M. J., Abdalla F. B., Banerji M., Adams R. G., 2010, MNRAS, 405, 987 (Cited on page 4.) Bonnett C., 2013, ArXiv e-prints (Cited on pages 4 and 149.)", "Bordoloi R., Lilly S. J., Amara A., 2010, MNRAS, 406, 881 (Cited on pages 6 and 138.) Bovy J. et al., 2011, ApJ, 729, 141 (Cited on page 123.) Bovy J. et al., 2012, ApJ, 749, 41 (Cited on page 123.) Brammer G. B., van Dokkum P. G., Coppi P., 2008, ApJ, 686, 1503 (Cited on pages 3 and 149.) Breiman L., 1996, Machine Learning, 24, 123, 10.1007/BF00058655 (Cited on pages 21 and 22.) Breiman L., 2001, Machine Learning, 45, 5 (Cited on pages 7, 21, 37, 81, and 120.) Breiman L., Friedman J. H., Olshen R. A., Stone C. J., 1984, Classification and Regression Trees, Statistics/ Probability Series. Wadsworth Publishing Company, Belmont, California, U.S.A. (Cited on pages 7, 16, and 17.) Brescia M., Cavuoti S., Longo G., De Stefano V., 2014, ArXiv e-prints (Cited on page 4.) Brett D. R., West R. G., Wheatley P. J., 2004, MNRAS, 353, 369 (Cited on page 5.) Brunner R. J., Connolly A. J., Szalay A. S., Bershady M. A., 1997, ApJL, 482, L21 (Cited on page 4.) Bruzual G., Charlot S., 2003, MNRAS, 344, 1000 (Cited on page 3.) Bundy K., Ellis R. S., Conselice C. J., 2005, ApJ, 625, 621 (Cited on page 12.) Capak P. et al., 2004, AJ, 127, 180 (Cited on page 12.) Carliles S., Budav\u00b4ari T., Heinis S., Priebe C., Szalay A., 2008, in Astronomical Society of the Pacific Conference Series, Vol. 394, Astronomical Data Analysis Software and Systems XVII, Argyle R. W., Bunclark P. S., Lewis J. R., eds., p. 521 (Cited on page 24.) Carliles S., Budav\u00b4ari T., Heinis S., Priebe C., Szalay A. S., 2010, ApJ, 712, 511 (Cited on pages 4 and 24.) Carnero A., S\u00b4anchez E., Crocce M., Cabr\u00b4e A., Gazta\u02dcnaga E., 2012, MNRAS, 419, 1689 (Cited on page 6.) Carrasco Kind M., Brunner R. J., 2013a, MNRAS, 432, 1483 (Cited on pages 4, 16, 64, 81, 116, 120, 122, 149, and 167.) Carrasco Kind M., Brunner R. J., 2013b, in Astronomical Society of the Pacific Conference Series, Vol. 475, Astronomical Society of the Pacific Conference Series, Friedel D. N., ed., p. 69 (Cited on pages 7, 71, 80, 84, and 168.) Carrasco Kind M., Brunner R. J., 2013c, TPZ: Trees for Photo-Z. Astrophysics Source Code Library, record ascl:1304.011 (Cited on page 4.) Carrasco Kind M., Brunner R. J., 2014a, MNRAS, 438, 3409 (Cited on pages 37, 120, and 167.) Carrasco Kind M., Brunner R. J., 2014b, MNRAS, 441, 3550 (Cited on pages 131 and 169.) Carrasco Kind M., Brunner R. J., 2014c, MNRAS, 442, 3380 (Cited on pages 5, 71, 149, 152, and 168.) Caruana R., Karampatziakis N., Yessenalina A., 2008, in Proceedings of the 25th international conference on Machine learning, ICML '08, ACM, New York, NY, USA, pp. 96\u2013103 (Cited on pages 16, 21, 57, and 80.) Cavuoti S., Brescia M., Longo G., Mercurio A., 2012, ArXiv e-prints (Cited on page 4.) Chisari N. E., Mandelbaum R., Strauss M. A., Huff E., Bahcall N., 2014, ArXiv e-prints (Cited on page 138.) Coe D., Ben\u00b4\u0131tez N., S\u00b4anchez S. F., Jee M., Bouwens R., Ford H., 2006, AJ, 132, 926 (Cited on pages 3 and 149.) Coe D., Moustakas L. A., 2009, ApJ, 706, 45 (Cited on page 2.)", "Coil A. L., 2013, The Large-Scale Structure of the Universe, Oswalt T. D., Keel W. C., eds., p. 387 (Cited on page 1.) Coleman G. D., Wu C.-C., Weedman D. W., 1980, ApJS, 43, 393 (Cited on pages 3, 76, and 78.) Collister A. A., Lahav O., 2004, PASP, 116, 345 (Cited on pages 4, 5, 24, and 149.) Connolly A. J., Csabai I., Szalay A. S., Koo D. C., Kron R. G., Munn J. A., 1995, AJ, 110, 2655 (Cited on page 4.) Cooper M. C. et al., 2012, MNRAS, 425, 2116 (Cited on page 15.) Cooray A., Hu W., Huterer D., Joffre M., 2001, ApJL, 557, L7 (Cited on page 158.) Cowie L. L., Barger A. J., Hu E. M., Capak P., Songaila A., 2004, AJ, 127, 3137 (Cited on page 12.) Crocce M., Cabr\u00b4e A., Gazta\u02dcnaga E., 2011, MNRAS, 414, 329 (Cited on page 159.) Csabai I. et al., 2003, AJ, 125, 580 (Cited on page 3.) Cunha C., Huterer D., Dor\u00b4e O., 2010, PhRvD, 82, 023004 (Cited on page 1.) Cunha C. E., Huterer D., Busha M. T., Wechsler R. H., 2012a, MNRAS, 423, 909 (Cited on pages 5 and 6.) Cunha C. E., Huterer D., Lin H., Busha M. T., Wechsler R. H., 2012b, ArXiv e-prints (Cited on pages 5 and 6.) Cunha C. E., Lima M., Oyaizu H., Frieman J., Lin H., 2009, MNRAS, 396, 2379 (Cited on pages 137 and 145.) Dahlen T. et al., 2013, ApJ, 775, 93 (Cited on pages 7, 84, 86, 90, and 91.) Dalal N., Dor\u00b4e O., Huterer D., Shirokov A., 2008, PhRvD, 77, 123514 (Cited on page 158.) Davis M. et al., 2003, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 4834, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Guhathakurta P., ed., pp. 161\u2013172 (Cited on page 13.) Davis M. et al., 2007, ApJL, 660, L1 (Cited on page 14.) Dawson K. S. et al., 2013, AJ, 145, 10 (Cited on pages 2, 11, and 14.) de Putter R. et al., 2012, ApJ, 761, 12 (Cited on page 1.) Debosscher J., Sarro L. M., Aerts C., Cuypers J., Vandenbussche B., Garrido R., Solano E., 2007, A&amp;A, 475, 1159 (Cited on page 87.) Dodelson S. et al., 2002, ApJ, 572, 140 (Cited on page 158.) Domingos P., Pazzani M., 1997, Machine Learning, 29, 103 (Cited on page 112.) Drinkwater M. J. et al., 2010, MNRAS, 401, 1429 (Cited on page 2.) Eisenstein D. J., Hu W., 1998, ApJ, 496, 605 (Cited on page 159.) Erben T. et al., 2013, MNRAS, 433, 2545 (Cited on page 14.) Faber S. M. et al., 2003, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 4841, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Iye M., Moorwood A. F. M., eds., pp. 1657\u20131669 (Cited on page 13.) Fadely R., Hogg D. W., Willman B., 2012, ApJ, 760, 15 (Cited on page 90.) Feldmann R. et al., 2006, MNRAS, 372, 565 (Cited on pages 3 and 149.)", "Flaugher B., 2005, International Journal of Modern Physics A, 20, 3121 (Cited on page 14.) Frank E., Trigg L., Holmes G., Witten I., 2000, Machine Learning, 41, 5 (Cited on page 112.) Freeman P. E., Newman J. A., Lee A. B., Richards J. W., Schafer C. M., 2009, MNRAS, 398, 2012 (Cited on page 4.) Fustes D., Manteiga M., Dafonte C., Arcay B., Ulla A., Smith K., Borrachero R., Sordo R., 2013, ArXiv e-prints: 1309.2418 (Cited on page 5.) Garilli B. et al., 2014, A&amp;A, 562, A23 (Cited on page 14.) Garilli B. et al., 2008, A&amp;A, 486, 683 (Cited on pages 14 and 15.) Geach J. E., 2012, MNRAS, 419, 2633 (Cited on pages 5 and 7.) Gerdes D. W., Sypniewski A. J., McKay T. A., Hao J., Weis M. R., Wechsler R. H., Busha M. T., 2010, ApJ, 715, 823 (Cited on pages 4, 24, 26, 31, and 149.) Giavalisco M. et al., 2004, ApJL, 600, L93 (Cited on page 12.) Gorecki A., Abate A., Ansari R., Barrau A., Baumont S., Moniez M., Ricol J.-S., 2014, A&amp;A, 561, A128 (Cited on pages 6 and 113.) G\u00b4orski K. M., Hivon E., Banday A. J., Wandelt B. D., Hansen F. K., Reinecke M., Bartelmann M., 2005, ApJ, 622, 759 (Cited on pages 57 and 158.) Graff P., Feroz F., Hobson M. P., Lasenby A., 2014, MNRAS, 441, 1741 (Cited on page 149.) Gregory P. C., Loredo T. J., 1992, ApJ, 398, 146 (Cited on page 87.) Gwyn S. D. J., 2012, AJ, 143, 38 (Cited on pages 13 and 14.) Hayes B., Brunner R., 2013, MNRAS, 428, 3487 (Cited on page 158.) Hayes B., Brunner R., Ross A., 2012, MNRAS, 421, 2043 (Cited on pages 138, 158, and 159.) Heymans C. et al., 2012, MNRAS, 427, 146 (Cited on pages 14 and 154.) Hildebrandt H. et al., 2010, A&amp;A, 523, A31 (Cited on pages 3, 12, 13, 38, 39, and 157.) Hildebrandt H. et al., 2012, MNRAS, 421, 2355 (Cited on pages 14 and 154.) Ho S. et al., 2012, ApJ, 761, 14 (Cited on pages 2 and 158.) H\u00a8ogbom J. A., 1974, A&amp;AS, 15, 417 (Cited on page 124.) Hogg D. W. et al., 1998, AJ, 115, 1418 (Cited on page 3.) Ilbert O. et al., 2006, A&amp;A, 457, 841 (Cited on pages 3 and 149.) in der Au A., Meusinger H., Schalldach P. F., Newholm M., 2012, A&amp;A, 547, A115 (Cited on page 5.) Jee M. J., Tyson J. A., Schneider M. D., Wittman D., Schmidt S., Hilbert S., 2013, ApJ, 765, 74 (Cited on pages 6 and 138.) Kinney A. L., Calzetti D., Bohlin R. C., McQuade K., Storchi-Bergmann T., Schmitt H. R., 1996, ApJ, 467, 38 (Cited on pages 3 and 76.) Kohonen T., 1982, Biological Cybernetics, 43, 59 (Cited on page 50.) Kohonen T., 1990, Proceedings of the IEEE, 78, 1464 (Cited on page 4.)", "Kohonen T., 2001, Self-Organizing Maps, Physics and astronomy online library. Springer-Verlag GmbH (Cited on pages 4 and 50.) Komatsu E. et al., 2011, ApJS, 192, 18 (Cited on page 165.) Koo D. C., 1985, AJ, 90, 418 (Cited on page 3.) Krone-Martins A., Ishida E. E. O., de Souza R. S., 2014, MNRAS, 443, L34 (Cited on page 4.) Larson D. et al., 2011, ApJS, 192, 16 (Cited on page 165.) Laurino O., D'Abrusco R., Longo G., Riccio G., 2011, MNRAS, 418, 2165 (Cited on page 30.) Lawrence R., Almasi G., Rushmeier H., 1999, Data Mining and Knowledge Discovery, 3, 171 (Cited on page 4.) Le F`evre O. et al., 2013, A&amp;A, 559, A14 (Cited on page 15.) Le F`evre O. et al., 2005, A&amp;A, 439, 877 (Cited on pages 14 and 15.) Leistedt B., Peiris H. V., Mortlock D. J., Benoit-L\u00b4evy A., Pontzen A., 2013, MNRAS, 435, 1857 (Cited on page 158.) Lemire D., Boytsov L., 2012, CoRR, abs/1209.2137 (Cited on page 134.) Lewis A., Challinor A., Lasenby A., 2000, ApJ, 538, 473 (Cited on page 159.) Lilly S. J. et al., 2009, ApJS, 184, 218 (Cited on page 15.) Lilly S. J. et al., 2007, ApJS, 172, 70 (Cited on page 15.) Lima M., Cunha C. E., Oyaizu H., Frieman J., Lin H., Sheldon E. S., 2008, MNRAS, 390, 118 (Cited on pages 4, 145, and 146.) Limber D. N., 1953, ApJ, 117, 134 (Cited on page 159.) Loh E. D., Spillar E. J., 1986, ApJ, 303, 154 (Cited on page 3.) Mallat S. G., Zhang Z., 1993, IEEE Transactions on Signal Processing, 41, 3397 (Cited on page 124.) Mandelbaum R. et al., 2008, MNRAS, 386, 781 (Cited on pages 6, 137, and 138.) Mannucci F., Basile F., Poggianti B. M., Cimatti A., Daddi E., Pozzetti L., Vanzi L., 2001, MNRAS, 326, 745 (Cited on page 3.) Matthews D. J., Newman J. A., Coil A. L., Cooper M. C., Gwyn S. D. J., 2013, ApJS, 204, 21 (Cited on pages 13, 71, and 78.) McMahon R. G., Banerji M., Gonzalez E., Koposov S. E., Bejar V. J., Lodieu N., Rebolo R., the VHS Collaboration, 2013, The Messenger, 154, 35 (Cited on page 15.) Melchior P. et al., 2014, ArXiv e-prints (Cited on page 15.) Monteith K., Carroll J. L., Seppi K., Martinez T., 2011, The 2011 International Joint Conference on Neural Networks, 2657 (Cited on pages 87 and 88.) Myers A. D., White M., Ball N. M., 2009, MNRAS, 399, 2279 (Cited on pages 6 and 138.) Naim A., Ratnatunga K. U., Griffiths R. E., 1997, ApJS, 111, 357 (Cited on page 5.) Newman J. et al., 2013b, ArXiv e-prints : 1309.5384 (Cited on page 3.) Newman J. A., 2008, ApJ, 684, 88 (Cited on page 170.)", "Newman J. A. et al., 2013a, ApJS, 208, 5 (Cited on pages 13 and 14.) Nicola A., Refregier A., Amara A., Paranjape A., 2014, ArXiv e-prints (Cited on page 158.) Oke J. B. et al., 1995, PASP, 107, 375 (Cited on page 13.) Oyaizu H., Lima M., Cunha C. E., Lin H., Frieman J., 2008a, ApJ, 689, 709 (Cited on pages 5 and 6.) Oyaizu H., Lima M., Cunha C. E., Lin H., Frieman J., Sheldon E. S., 2008b, ApJ, 674, 768 (Cited on pages 4, 24, 31, and 149.) Parkinson D., Liddle A. R., 2013, Statistical Analysis and Data Mining, 6, 3 (Cited on page 87.) Percival W. J. et al., 2010, MNRAS, 401, 2148 (Cited on page 2.) Perlmutter S. et al., 1999, ApJ, 517, 565 (Cited on page 1.) Planck Collaboration et al., 2013, ArXiv e-prints (Cited on page 1.) Reddy N. A., Steidel C. C., Erb D. K., Shapley A. E., Pettini M., 2006, ApJ, 653, 1004 (Cited on page 12.) Reid B. A. et al., 2010, MNRAS, 404, 60 (Cited on page 2.) Riess A. G. et al., 1998, AJ, 116, 1009 (Cited on page 1.) Rokach L., 2010, Artificial Intelligence Review, 33, 1 (Cited on page 84.) Ross A. J., Percival W. J., Brunner R. J., 2010, MNRAS, 407, 420 (Cited on page 1.) S\u00b4anchez A. G. et al., 2013, MNRAS, 433, 1202 (Cited on page 2.) S\u00b4anchez C. et al., 2014, ArXiv e-prints (Cited on pages 3, 15, 144, 145, 149, 150, 161, and 169.) Sawicki M., 2012, PASP, 124, 1208 (Cited on page 3.) Schapire R. E., Freund Y., Bartlett P., Lee W. S., 1998, The Annals of Statistics, 26, 1651 (Cited on page 24.) Schlegel D. J., Finkbeiner D. P., Davis M., 1998, ApJ, 500, 525 (Cited on page 13.) Schmidt M., Lipson H., 2009, Science, 324, 81 (Cited on page 4.) Seo H.-J. et al., 2012, ApJ, 761, 13 (Cited on page 158.) Sheldon E. S., Cunha C. E., Mandelbaum R., Brinkmann J., Weaver B. A., 2012, ApJS, 201, 32 (Cited on page 6.) Sheth R. K., 2007, MNRAS, 378, 709 (Cited on page 6.) Simet M. et al., 2012, ApJ, 748, 128 (Cited on page 1.) Smith R. E. et al., 2003, MNRAS, 341, 1311 (Cited on page 159.) Strateva I. et al., 2001, AJ, 122, 1861 (Cited on page 33.) Strauss M. A. et al., 2002, AJ, 124, 1810 (Cited on page 10.) Tegmark M. et al., 2002, ApJ, 571, 191 (Cited on pages 158, 160, and 164.) Thomas S. A., Abdalla F. B., Lahav O., 2011, MNRAS, 412, 1669 (Cited on page 158.) Tipping M. E., 2001, J. Mach. Learn. Res., 1, 211 (Cited on page 149.) Treu T., Ellis R. S., Liao T. X., van Dokkum P. G., 2005, ApJL, 622, L5 (Cited on page 12.)", "Trimble V., 1987, ARA&amp;A, 25, 425 (Cited on page 1.) Trotta R., 2007, MNRAS, 378, 72 (Cited on page 87.) van Breukelen C., Clewley L., 2009, MNRAS, 395, 1845 (Cited on page 6.) Vogeley M. S., Szalay A. S., 1996, ApJ, 465, 34 (Cited on page 158.) Wadadekar Y., 2005, PASP, 117, 79 (Cited on page 4.) Wang D., Zhang Y.-X., Liu C., Zhao Y.-H., 2008, ChJAA, 8, 119 (Cited on page 3.) Wang W.-H., Cowie L. L., Barger A. J., 2006, ApJ, 647, 74 (Cited on page 12.) Wang Y., Brunner R. J., Dolence J. C., 2013, MNRAS, 432, 1961 (Cited on page 138.) Way M. J., Foster L. V., Gazis P. R., Srivastava A. N., 2009, ApJ, 706, 623 (Cited on page 4.) Way M. J., Klose C. D., 2012, PASP, 124, 274 (Cited on pages 5, 7, and 68.) Wirth G. D. et al., 2004, AJ, 127, 3121 (Cited on page 12.) Wittman D., 2009, ApJL, 700, L174 (Cited on pages 123, 138, and 141.) Wolf C., 2009, MNRAS, 397, 520 (Cited on page 3.) Yip C.-W., Szalay A. S., Carliles S., Budav\u00b4ari T., 2011, ApJ, 730, 54 (Cited on page 40.) York D. G. et al., 2000, AJ, 120, 1579 (Cited on pages 2 and 10.) Zhan H., Knox L., 2006, ApJ, 644, 663 (Cited on page 2.) Zhang H., 2004, in Proceedings of the Seventeenth International Florida Artificial Intelligence Research Society Conference (FLAIRS 2004), Barr V., Markov Z., eds., AAAI Press (Cited on pages 80, 112, and 113.)"]}