step outside of that that you can leverage metacognition to think about your thinking and Deconstruct and decide what you want to think about and start focusing on that and create an entirely different version of yourself that has new Elevated feelings that's over on the side of the positivity empowering yourself I think it's really incredible and he gets deep into the mechanistic stuff Which I love you guys will not regret diving deep into this man's world. I think you will get some incredible revelations All right, if you haven't already be sure to subscribe and until next time my friends be legendary. Take care


[Music] oh hello everybody um I'm really excited to host this panel about the extent to which mobile apps can contain the kovat 19 crisis as you all know we are developers is the largest community of software developers in Europe and in doing so we like to produce exciting content for developers which is not just tech content but also content that affects us as a society in a political context and from an economic standpoint at this point I would like to thank fit for internet Austrian organization that aims to digitalize the

What extent can mobile apps solve the COVID-19 crisis?


Austrian population for collaborating with us and making this panel possible and I'm excited to introduce to you the three guests of honor we have today joining us in this panel first and foremost I would like to say hello to Antonella my Hoffler special advisor to the federal Chancellor of Austria head of the strategy unit think Austria co-head of the future operations clearing word that has been set up to plan the host Perona era she's a long-standing senior partner and managing director and now senior advisor


of the Boston Consulting Group and a member of various supervisory boards welcome Antonella thank you next up I'd like to say hi to Tomas inertness he's the CEO of CEO of ions Telecom our group the largest telecommunications provider in Austria and one of the largest in the sea region hello thanks for having me I told us and last but not least the person I'm going to begin this talk with me a secular country managing director of Accenture Austria it's great to have you here hi and thanks for the invitation so Michael and


jump right into the topic so Accenture developed the Austrian Corona app how did this project come into being when we received the car back area phytic at the head of the Austrian red course in regards to fight in Kuwait on March 9th and we thought back then together that first of all technology is an ally in our fight against this new disease so we thought it's a goodness is to use an app in order to help fight the spread of Corona and therefore we quickly set up a team and within roughly two and a half weeks we were able to design test and


ROI and f2 the App Store's first with say a very limited set of features but already productive in order to get first insights from production and usage and then we continually improve the app up to the status where we are now where the automatic handshake is working and where we are waiting for Google and Apple to provide further features in order to further improve i'b what were the challenges in developing this app and how are you able to develop it so quickly seeing that Germany is still trying to launch their up quality the challenge is


better more or less overwhelming first of all the technology we have mobile phones actually not suited for this purpose so bluetooth is only able to mesh you're only able to measure the signal strength for Bluetooth you're not able to measure distance so first of all we had to overcome the technical problems associated and they are still not solve to be very clear I mean the technically say workaround is first of all when measure or at the contact which is relevant for this disease is combined of two parameters which is time and


distance and time is even more than distance if you listen to what scientists in the medical field say and the recommendations by the World Health Organization so time is an important parameter distance can only be approached by the signal strengths of Pluto that's the first issue the second point is that especially with the iOS operating system so with our ever iPhones bluetooth low in low energy mode cannot receive blue other Bluetooth signals in the background so the app has to be in the foreground which is then of


course very battery consuming and which naturally leads to a limitation of in the usage with iPhones in the current state now as a plan could have made that alliance has built have built an alliance that problem shall be overcome with in the next weeks we hope so that by June we will have the major technology limitations currently in the operating systems we have we are able to cope with them so that's the that's one of the that's a technology view and of course we had significant obstacles as well in saying everything around so


first of all data protection was a very important goal from the very beginning and of course the inter protection also is a challenge from a technology point of view so we had to implement a lot of additional actions and a lot of mechanisms in order to to ensure that the privacy is is guaranteed within the app and that's an of course an ongoing process as it is with every security measure you take you always learn you always improve and that's where we invest a lot as well so that's the second point the third


point is then of course we want to have the app work as widely as possible so ideally within the whole world at least within Europe which is currently not possible as there is no standard protocol available for the European Union for example so we need to find standard first which can template be deployed within the whole European Union which is currently not the case so we are working on that standard as well we try to bring our own expertise but also talking and listening to others and their expertise to find a common standard across Europe and


the last point of course we got a lot of attention actually we were a bit overwhelmed by the press and and media response to that app we were truly surprised that it's such a I would say politically or a sensitive topic we thought it's totally natural that we use technology in order to fight the disease okay you talked about the topic of data protection and of course for us as a huge issue so um where's the data stored and how do you guarantee that the data is safe and secure well first of all in regards to data protection we had a lot


of external reviews very thorough reviews by NGOs I mean as in open we have one of the lead leading NGOs for data protection none of your business no max James and his team is one of the leading data protection organizations in Europe at least if not in the whole world so thought we gave those guys in very early stage look at the full source code without any limitations and they they made a review independently of course so they weren't paid by us they did it by themselves on their own and they came up with a lot of


recommendations which we implemented and we are still in exchange with them so they have access to the source code we we inform them on the changes we do and keep them posted so to ensure that third parties can take a look at it in the meantime we've also open sourced the whole code to the public so everybody can take a look at it at github from a data perspective that's the most important point because it was so intensively discussed and publicly discussed there is the only contact their contact data is only stored on the


mobile phone itself so nobody can access the contact data by any means and it is stored anonymously or technically correct Stud anomaly so you can't access any personal data in this app we can exclude that so there's one point where we need to exchange information through a server which is when we notify people so if somebody it wants to issue a warning or a positive result of her of a cuvette test then in this case he has to give his telephone number in order to receive transaction number to have something like or to have


a security mechanism so that it's that to make sure that it's your mobile phone which from which the information was sent and also when sending this this when issuing this warning of course the notification needs to be sent to a server but again this is done without the detailed contact data but also only using anonymous data how many users have downloaded the app so far we have roughly 600 K so 600,000 downloads and yeah that's where we are of course we hope for more and we hope that the usage will significantly increase once the


improvements by a cookie and April are available within our app ok and one one more question on that have you considered incentivizing the use of the app so for example the EU politician axial force he suggested some ways how you could positively incentivize people downloading and using the app and be sure that we have tons of ideas on that but it's a very sensitive topic I'll ask Antonella my ideas and how to incentivize the use of the app but there are many good ideas and for example I mean to give you two small ones at least


we implement the recommendation function so actually this week's of just one or two days ago with the new version we implemented a couple of additional security mechanisms and now you can also recommend the app to your friends so that's a very small feature and there are first to answer vada yes one is to have a campaign start more or less the whole society and that is something we hope that the Austrian Red Cross will put further effort into this okay well thanks that's that's really interesting information I'd like to continue with


you Antonella could you tell us about the work that you're currently doing with pink Austria and with the future operations clearing board with regards to on the one hand containing the coronavirus and on the other hand ensuring that the austrian economy hits the floor-- running once we're in the posts covered 19 period so thank you for that I will focus on the future operations clearing board because that is looking at exactly the topic that you are asking for so as you can imagine for us it is paramount to be able to manage


the so-called tans face as we all know in the meantime you know after the hammer phase to be able to manage the dance phase in the best possible way so that we can get to a quick restart and we are all very much aware that we have extremely strong impact on the economy overall and on the life of everybody so what we look at that the future operations clearing board is at four different areas which we try to you want to triage to try to combine so the first area looking at the management of the health area of the health system and for


that I will come to that for that the app is a part of the solution it's not the whole solution unfortunately it would be nice if we just had one app that can solve everything but we need to find a way to track trace and test in significant numbers to be able to manage this containment so that is the first area is managing the the infection and managing it in a very differentiated way and we'll come to that mode EP the second area just for for your information the second area is what recall the provisioning of basic


services you know so looking at the logistics chains looking at the food chain at other critical areas like medicines how are they impacted by possible lock ups in different countries in different companies and so that is something that we are we are analytically trying to capture and to see whether we have significant trigger point and breakup points in the overall supply chains that's the second area the third area is economy at large and its economy it's the financial system it's the tourism system it's anything which


has to do with with the workforce with with the impact that it has on the economic activity that is where the focus over time will certainly become more important because that is where the full impact is is visible already now and well we need to create the prerequisites for a quick restart and also for I would call it a reset because we will have different areas well which will become more important and certain areas which will be which will be impacted in a sustained way let me put it this way and you know it's a


certainly we can go in more detail and the last area is the psychosocial impact needless to say people are impacted by what is going on there is a great angst in people there is a lack of there was a lack of security at the beginning particularly the more we manage the particular health element properly the more we will have you know the less you know the burden is on on people and particularly there let me highlight one point what we see is that certain groups of the population are more impacted than others and particularly women and women


with small children of much impacted by that so we are tracking that and trying to make it possible to take the burden from this from these groups and that many others were really impacted just let me come to the first area because that is where we feel that technology is definitely part of the solution actually apps can help in many areas but particularly you know the the t-t-t-that testing tracking and tracing is one of the critical areas so making sure that we have enough capacity to track and trace when we open the borders when we


have more tourists where tourists coming in where we have a stronger mixing of population I think is very important and that is why we feel that it's it's critical to have an app that is very much accepted by the population so what do we mean by they're much accepted it has to fulfill three three key criteria it needs to fully comply with the privacy expectations of the population and that we are very clear and I think you know Michael was very explicit you know we have very clear guidelines with it needs to be privacy compliant and it


needs to be accepted second thing it's usability now on the usability we all agree that it was a minimal Viable Product what was introduced and I think it's a it's a it was a great proof of speed of bringing it to the market it's still not as function as you know as performing as as we would love it to be the more performing it gets and more usable it gets the higher the acceptance and I think that then people will like to to use it the third thing and that is also mission critical is interoperability because clearly why do we need it we


need it because we want to make sure that we are able to act once we open the borders so these are three key elements but let me come to the fourth essential element its individual responsibility we will never get control of this health crisis of this infection if people don't take the individual responsibility of making sure that we manage this in the proper way and I do not think that we can forever managing top-down I don't think that's the that's what we want and it's also makes no sense it's not what we what


were used to and it's also not going to work in general we need to enforce you know enforce a sense of responsibility if anything but not enforce tools or other stuff you know and that is what we that that is what we totally think and that is what we are clearly factoring in in our considerations in the future operation spot under which preconditions taking us back to the topic of that under which preconditions does the corona have fulfilled its purpose I think it's it will fulfill its purpose if it really works you know in a


hassle-free mode I mean usability as I said is essential and you know I think that you know there are non-debatable switches privacy clearly that's a known debatable we don't need to discuss about that but the usability is the essential thing it needs to be it needs to be working you know easy in an easy way and clearly you know and I think you know we could we could start to see a broader a broader usage once the usability gets better but then interoperability is then the the other key problem that's that I


think is really the it's the to maker or breakers because then we can we can make sure that we move around and we are we have a sent not only a sense but a real tool for individual security and I think that's and for individual control for self control I think this was moving to a responsible behavior - self-control is essential ok which what percentage of the Austrian population has to install and use the app in order for it to be effective I don't think that it's about broad percentages of the total population I


think that we need to have this particularly you know brought in in broader usage where where the risk of super spreading is highest so I think that we need to be you know we need to see that the risk of being infected of encounter of a infected person is very different if I'm somewhere in at the if I'm you know someone Berglund or if I'm at the border of German Austrian border and Tirol so I think it's very or in Vienna which has a very high you know confluence of a lot of people so I don't think that we should


aim at broad percentages let us try to look at how if it's used where its most needed and that is where the encounters of different of groups from different people more more expected to happen mm-hmm but I mean I did a little bit of research and there's like experts or experts suggesting that 60 to 70 percent of the population should be should have installed and be using that prefer to be effective from what Michael just said were right now around 7% a little bit a little bit below 7% don't you feel that we need to get a


much higher rate of usage in the next month in order for the project to work no absolutely I think that we need to increase the use of the users rate we need to increase it and and if you and it were clear you know about that it is about making sure that for example companies start to introduce you know to recommend the usage of an app and the people in a company which are usually working together that they also ask their the people that they are working with and and there you know other people that they are interacting


with to also use it so that there is a bit of a control but I think you know it's not abroad I mean if it's if 60 to 70 percent of their awesome population is expected to use it then we can forget about it you know I think we should not define irrational high bars we should introduce sensible D averaged targets and make sure that it's used where the risk is highest okay now obviously the whole topic of the app has created quite a media frenzy a bit of a heated discussion on relevant or less relevant topics and actually might have been


taken out of context one of the topics that the the whole discussion focuses on at least what I've been reading on is that there is kind of two different aspects of the whole situation the one aspect is saving lives because let's face it it is at the end of the day a virus that causes a lot of death and saving life is the priority and a responsibility that that we need to take care of but on the other hand we have the topic of freedom of choice democracy and what do you think is more important I mean what I mean freedom of choice


democracy is always more important than anything else it's there that's more important than anything else but I think yet we need to be I mean the if if we look at the discussion I think that we are mixing a lot of things you know you you rightly said it is a matter of balancing the needs for for health control with other needs and given that we have a very very performing data privacy regime in in Europe we are in a match that situation because we have the possibility to create things that are totally compliant with this data regime


which is extremely essential it's it's totally fitting the the belief in democracy and in individual free choice that is you know a pillar in Europe so I think that's as I said it's a non debatable about that but then we need to be also very clear about what does it mean if people you know willingly and the mind the discussion about these apps you know because of I don't know what reasons because it's makes no sense but at the same time the same people have a broadly used commercial apps and have profit sharing agreements and and then


which you know the user of their services don't even know about and and these commercially commercial companies fully capture the data not for protecting the health of anybody apart from the economic business health of the company so I think that we need to be we need to have a serious discussion about what is the right governance for for this kind of health and and life-saving apps clearly I mean compliance with the privacy rules is essential and and then there are some elements that I feel a very important like making sure that


open technology and also that you make sure that that they are used in the spirit of individual responsibility I think that's essential okay you mentioned previously that one of the four pillars that the future operations clearing board is working on is the topic of the economy the financial system and tourism so since Austria is heavily dependent on tourism as a pillar of our economic prosperity how do we handle how do we tackle the challenge that tourists are entering the country and they're not using the corona app how do


we what we can do if we have look I think you know the tourists the individual tourists comes to Austria because he wants to have a fantastic time he wants to be nature wants to experience our beautiful country so that is what what they want they don't come primarily to either to get into quarantine you know or to get or to infect others so I think that if we have if we make sure that we get what Michael has promised to deliver which is a very usable very performing app that helps them control I think that everybody will


happily download it and make sure that that he or she is neither risk to others but also that others are not a risk for him or her and I think you know that is what we feel is very important and clearly one can it's our duty to make them aware I think that Michael was mentioning the fact that we need perhaps to to have a more stronger communication campaign at least we need to make them aware that there is a stop Karuna app you know if they don't know about it it's difficult they come from countries where the the app is not yet there so


how should they know that we have an app I think that we should make them aware we should invite them to use it I think that's that's quite important and and then make sure that people really take their individual responsibility seriously so are you also working on measures to communicate the availability of the app to tourists coming to Austria not not directly I mean this is something that red cross has on top of the mind and and is working on what we are looking at is looking at the processes and looking at


the prerequisites for TTT for tracking testing tracing so that that is in that is really available in the right way in Austria so that when people come in they can experience experience a really very safe country if we're not one of the safest countries in Europe I would like to touch upon something that Michael sets in the beginning obviously Corbett 19 is causing a global crisis it's affecting nearly every country in the world so do you think it makes sense to treat a global problem on a local basis I think it's I think you need to treat


the global problem at a global and at a local level if you want to tackle it and I think that's true for most of the huge problems that we are facing not only for Corbett but also for the environmental problems we need we do not have the right governance for many of these problems because in a sense you need the global concepts to to fight the problem but then you need really to break it down on an individual and on a local level you cannot enforce the top-down it needs to come from the bottom up and the better the concepts that are developed


on a global level the more they will be taken up and as we know we live in very different types of countries and every country will have their own way of communicating with their citizen and making sure that their citizens are made aware about what solutions are best so clearly we saw the example of China we saw the example of Singapore we see the example of Taiwan and we see the example of Austria or Italy they are totally different that we need to be mindful about what is working in our specific cultural context and political context


and I think that that's very important so yes it's it's a it's a Savola at South in German you know it's a clearly you need to work on a global level but you need us to work on a local level and are there activities to work on a europe-wide corona apps because Germany is working on one Austria already published its app are there activities going in this direction there is a lot of activity on a European level but I think that at the moment I think that Michael can probably say more to this bilateral working bilaterally is


probably one of the other routes but you know I think that we need to start somewhere and having having a working app somewhere is a good starting point okay thank you very much Tomas I would like to continue with you also focusing on a topic that has cost for heated discussion so SEO of iron steel camasta which is one of the largest telco providers in the CEA and region and in Austria you've provided the government with anonymized data on the movement of groups and individuals in collaboration with in vain IAM which is a spinoff from


the Technical University of god´s some insights on how this initiative has contributed to the management and the containment of the current crisis well for sure thanks for the invitation once again first of all let me start with saying that when this crisis started and actually the first lockdown measures were implemented in Austria middle of March we somehow went through a similar experience as marketed because when we were discussing internally how can we sa technology company contribute to the handling of this crisis and when we


actually went into discussions with the crisis management of the Republic of Austria very soon after we got some significant media backlash where maybe some fair concerns were addressed but much of the issues raised were really based on a fundamental lack of information and when people said we cannot use big data as a means in tackling these crisis I must say I was somehow concerned if not to say shocked to see that the to see the amount of resistance versus using facts data evidence to take valuable decisions in technique this crisis and I


was shocked to see that people don't understand that not every use of data in handling these crisis does necessarily imply an infringement of fundamental rights such as the privacy let me explain what we do in a mobile network you always need and have so-called signalling information that is the information which you need to set up for example a call of a mobile phone which you need to know to which sales site mobile site a phone has to connect itself it's about controlling the network and in the signaling traffic we


are already using an IDE called the MZ that's the international mobile subscriber identity which is in the systems attached to name of a person but in the signalling systems it's only a number of any given cell phone attached to the network we use this information and we immediately anonymize this because we attach to each of these energy numbers in a normalized hashtag which is by the way exchanged every 24 hours so also we don't have any historical individual patterns of customers the collaborate with a spin-off of the Technical University


cards called indium which have very sophisticated algorithms in using these information to get insights on the mobility of people when I say people and referring to anonymized aggregated clouds of people's with a minimum website of 20 people or more we do not have information on the any individual movement of people and this is by the way not a use case of this information and that's another learning that people really tend to mix up the different use cases in handling this crisis but we were interested in and what we're


providing to the crisis part of the Republic of Austria is information of the kind of how how many how many people less have moved for example in the inner city of Vienna how many people have not moved beyond a parameter of 10 kilometers from their assumed and we cannot only assume because we don't have this information from the assumed whole because what we wanted to understand and what proved to be very at at least what the crisis management tells us what proved to be very helpful to the crisis management is to understand to the


extent of reduction of mobility when the lockdown measures were implemented and the extent of additional movement with every step of lifting these lockdown measures the solution is fully gdpr compliant it has been tested by external auditors even the head of the board of the Austrian so-called attitudes behind the regulatory body taking care of the implementation and enforcement of gdpr has confirmed that the solution is gbbr compliant and we believe there is a very valuable piece of information in assessing the effectiveness of the


lockdown measures now having said this I fully agree with what - Michael and Antonella said before I do believe that especially as technology companies we do have a responsibility to provide technology and we do have a moral obligation to provide technology which helps addressing these major crisis and even if the solution we are providing would constitute an infringement of fundamental rights I really want to point out affected in all these use cases we are using technology to lessen the impact of the infringement on


fundamental rights because every day we are subject to much more material infringement of our fundamental human rights such as the right to live the right to live a healthy life the life - the right to an adequate standard of living just think about the existential threats to many businesses to many individuals which is at risk every day we are speaking every day think about the freedom of movement which has been inhibited all across the globe significantly or just the right of peaceful assembly in the using


technology in order to get back to at least more normal way of life with less infringement of fundamental rights every day and I think it's a moral obligation actually to do this and we should be asking ourselves what we would be asked if he would not prove up I think the last sentence was cut off could you distribute that if you could not provide well if we are asked whether its moral morally defendable to use technology we also have to ask the question the other way around is it more about just morally justifiable not to


use technology which we have available to take it is crisis and I think that's one of the learnings also of this crisis not only when we talk about apps or mobility insights without technology we would not at all be in a position to handle the crisis we are handling it today think about how we would deal with the situation 10 or 20 years ago or what happened a hundred years ago in the Spanish Flu we would not even understand the DNA of the virus today we will not be able to test it within hours we would not be able to continue working


in our home offices we would not be able to educate our students and pupils at home he cetera et cetera et cetera so I understand and I take very seriously all the concerns also with regards to privacy within when we use these pieces of technology but I believe we really need to calibrate also our approach to technology and really embrace the fact that without technology we would be in much better shape in handling this crisis but this this again brings us back to this topic of the ability to save lives but at the same


time somehow infringing the freedom of choice of the population correct and that's why I believe and I think Antonella has beautifully described the important criteria when using these these tools and first we need to be clear that we should not mix abuse cases and there is a fundamental difference between what we do when we do general assessments on the amount of mobility of large groups of persons versus personalized even if it's anonymous but still personalized contact tracing and when we look into international examples there are other


examples around of voluntary or imposed Tracking's of quarantine measures which is again another topic but I do believe that in the public debate people tend to put this all in one basket and that's an issue that's a real to be frank an education issue I don't believe that with these measures you avail advice to do it in a top-down approach in non-voluntary approaches in turn Ella has described it because especially in these crisis in in crisis mode and we also know it from our businesses it's about changing behaviors


of large masses of people and that's like a large transformation project in a company it does not work top-down you need some enforcement enforcement measures for people who come let's say completely off track and to completely disregard all let's say all reasonable behavior but for a large amount of people you need to get buy-in in the buy-in requires trust and looking forward Trust will be extremely important in handling the crisis also when we talk about handling the economic crisis and we need more trust in


institutions not less and we will get trapped only if we consider design principles like the privacy like using anonymous thought data as long as as much as we can use it by using decentralized solutions by storing data carefully and by especially feeling on the voluntary participation of people and not on the obligation enforcement um I did a bit of research on how you collect the data how you analyze the movements of groups of 20 plus individuals is it correct that you're solely tracking the data of mobile


phones in the telecom master Network yes that's correct however and by the way this use case has been around for some time we also use this data for example to measure how many people are traveling on the train because it's also quite a challenge because train you don't have you know it's not like on the plane where you have dedicated seating or you have physical barriers to enter the Train but as we do know our market share is in given areas and as this model has been calibrated also with external health we


have at least very high confidence to calculate on the entire basis would it make sense to collaborate with the other two telco operators in Austria to get a more granular they're granular database well we would be open to do so however I do believe that the data we are able to provide the sufficient is sufficiently concise in order to draw similar conclusions today already ok I have a bit of a another example I don't agree with the approach that was taken but if we look at what the Israelis did in mid March of this


year it became public that Prime Minister Binyamin Netanyahu authorized Israel's Internal Security Agency Shin Bet to use the location data of cellphones to help contain the coronavirus so what they did is they actually track the movements of individuals who tested positively for Kovach 19 and identified the individuals who came in touch with these infected people and these individuals receive text messages informing them anonymously of the fact that they had been in touch with an infected person just from a


technological standpoint would you be able to do this would you be able to build a system that allows us to basically cover the entire user base of the ions tilaka monster network anonymously in a safe and private method in order to effectively have the same result as what Michael and his team at Accenture had created with the corona no clearly not based on the technology we using when we do the mobility insights we are using more or less triangulation out of the mobile network we are not using for example any GPS data from the


cell phones and with that we can see if people move let's say from one community to to another one or if they move from Vienna to Lynch or whatever but we cannot sufficiently assess whether I would be sitting in the Train next to you or you are sitting at the beginning of the Train and then actually also where we are not even interested in the information and we don't have to technologically I set access to that information I think that's more a topic of the very front okay but combining your data with GPS and Wi-Fi data would


that be an option and perhaps with with Michael's app for us it's not an option because I do believe that the app is a very smart approach I do have the upon myself on myself I agree with them Antonella and and Michael I think it's quite a long way to go in order to have it let's say fully functional and fully performing but we also have to consider that I think in Austria we have the advantage of having head very early start we have the advantage of having from my non expert view the right setup from the beginning


with a decentralized approach with an voluntary approach with a very transparent approach based on open source so I think we are going in the right direction and I have quite some confidence that at the end of the day it will help us tackling this pandemic and we shouldn't forget this pandemic from all what we hear and know will be around for months if not years and we have now to prepare in order to handle what comes in months and years right so we are not at the end of the tunnel I think we are at the beginning of the tunnel and we


have an early start in using technology I think the app and the use case of Traken traces one thing and what we can do in the network in assessing the amount of mobility is another use case but we should mix it - thanks Tomas a question to all of you what are the international kind of best practices that you've seen when it comes to using technology in order to tackle the library if you want I can start because this is what we have been looking into a lot I think it depends you know clearly you can find a lot of interesting


examples particularly a shoe where they had already the experience the previous Czar's experience and that has has sharpened their view on how to manage pandemics in general and also sharpen the view on how to make the best use of of technology so I think that you know what we saw in South Korea they had both and I think that Thomas was also mentioning it we speaking always of apps but there are so many different apps you know there are symptoms checking apps which are very broadly used in Korea symptom checking apps are very


interesting because they are a first line sort of defense if you want or a first line of information that everyone can take before going into a system that is certainly easily burdened of going into fully testing and tracking and tracing so symptom tracking apps are broadly used in Konya and I think that is something that is super useful then they have tracking and tracing Epson use but mainly also parent time surveillance apps which are used also not only in Korea but also used in Singapore and also used in Poland you know that we


have quite exciting interesting examples you mentioned the Israeli example I think that it's we have to see Israel is a country which is on constant high alert and I think this being constant high alert allows allows to interact with the population in a very different way you know so we are not in a constant high alert as we have seen we need to create the sense of alert so that people can act you know responsibly and so what is working there there are very interesting tools there but they would not work here so that is the other thing


what we have been also looking at is the potential use of immunity of the immunity apps which allow you to fully capture your testing process and the tests result on your app so that you have it easily usable I think this would be a very interesting solution particularly on a European level or on a global level but part you know a European level because then you can really travel and make sure that you you know you minimize the risk so that is also something that that we've seen and then they're clearly your traffic


management apps broadly speaking I mean anything which allows to book things upfront viewing the menus in restaurants without having physical menus be presented I think it's something that is being discussed anything which is sort of touch free which are touch free solutions touch free solutions beyond you're attaching your own phone I think that is something that are being used and I think that if we look around we have some countries which have been very active in working with that Australia has introduced a very good tracking and


tracing app Singapore as use a combination of these apps so these are the countries but with that we look into but then we need to sort of factor it in that it should be used in Europe okay would you like to add anything Tomas or Michael I think there was already a very comprehensive summary of technologies available by Antonella but a really one would like to reiterate is that I think actually with the two solutions used in Austria you have mentioned your Italy at least leading one of the leaders in Europe which the


stop Corona app with I think as I said earlier a which was taking technology as well as in timing or those on our side I must say after the first I should have put his surprise in the public attention our initiative has a stake we got a lot of international requests actually also from other operators who by the way have similar solutions in place but many of them do not prove a creators hours to us and here and by the way I was also invited to a court with a commissioner Kato with the ambition to have similar use cases on


European level because I agree with what was said before interoperability in all what we do will be key because especially in Europe with our small structures we need to open up the borders as soon as possible and if technology can help you it's for sure - so I think we in this respect we are also leading to take you within Europe in let me just add one thing thanks Thomas also for the acknowledgments that we are leading in Europe I think helps in speaking more general is health is one of the most under penetrated areas


for the use of technology for good so I truly hope that we see a push towards the use of technology in health chain in in a more general way such as using telemedicine for to give one example and and for a couple of other use cases where we could use technology in a far better way than we do today based on say resistance in society where technology really could make a difference and that's what at least I hope that we can catch up on that and help to improve the overall situation of the health systems with technology but


may I just build on something that told us I said before I think the critical thing is really making sure that the population appreciates truly the value of technology in liberating us in making us freer in giving us our individual freedom of movement freedom of speech freedom of everything you know and the technology can help absolutely but we need to probably invest significantly more educating people because there is a lot of mixing up things you know and in general suspiciousness big suspiciousness towards certain certain


you know solutions that are very very important for for the individuals and for the society for the collective solution of this of the problems and i think that that's where we we jointly need to need to act and make sure that particularly you know the younger generations are totally you know just have a full understanding of where the risks are and where the risks are not and that they develop a good sense of trust and mistrust and not just you know just ad averaged mistrust about certain people and total trust towards others


and i think and going down to naivete yeah and i think that's what we need to be careful about okay well i think we're we're solely coming to an end i would really like to thank the three of you for your great contributions internet i think it's great that you're taking such a helicopter perspective and tackling so many topics at the same time also thanks for for clarifying the the data protection and the data privacy aspects that i think are very important to the population and that you've also demonstrated and we can confirm this


hi there today we'll look at big self supervised models are strong semi-supervised learners by ting chen simon Kornbluth kevin sworsky muhammad nur Uzi and Geoffrey Hinton of Google brain so this paper on a high level it's also known as Sinclair v2 demonstrates that if you want to do semi-supervised learning that you're very well served by starting out with self supervised learning and then doing fine-tuning much like NLP models do rather than the kind of semi-supervised approach that image image tasks had so far and they present

Big Self-Supervised Models are Strong Semi-Supervised Learners (Paper Explained)


this same clear v2 which is an improvement over the same clear approach to self supervised pre training and they demonstrate it outperforms a lot of the baselines alright so if you like content like this don't forget to share it out and leave a like and tell me what you think in the comments so this paper um it sort of is kind of a club together thing of different of different things so they present this new method like simply same clear v2 which is a modification of sim clear and we'll go over that but they also try to


make like a and a scientific claim namely that the that somehow bigger models are better for this pathway of learning and we'll try to untangle all of these things so first of all we're in the semi supprised learning regime right here semi-supervised basically means that you have a data set and you only have labels for a part of that data set so this could be like here at the bottom 10% or so because labels might be expensive to get and so you only have a few of them but you have much more data that's unlabeled now sometimes this


problem is formulated as this here is your data set and then this here is like a different data set but one that's close enough such that you can learn from it and that's usually in in NLP you'll have your data set is like a sentiment classification task but you have all of Wikipedia that is not labeled but it's just text so you can sort of pre train on it in this case we'll be in a situation where will artificially construct a small data set so this entire thing here is going to be the image net data set and this right


here is going to be our labelled portion like we have labels now usually one has labels for image net as well but we artificially restrict ourselves to simulate a situation where we have lots of data and we only have a fixed budget so we can only because to obtain labels often times you have to ask humans right to label images and let's say we're a company and we've collected this big data set but we only have like maybe 500 bucks on Amazon Mechanical Turk and we only managed to get a very flat aset labeled now we're in the in the regime


of semi-supervised learning ok this is slightly different from what NLP does and as I said in NLP usually assume you have different data sets the large one being the different distribution and in this semi-supervised regime you often assume that it is actually the same data distribution but you only have labels for some of them but there should be a fair bit of overlap between the two things so oh I've recently made a video about open a eyes image GPT that kind of goes into the into the same direction as this work right here that basically says


pre training on unlabeled data like this whole data set without the labels can be a very good pre conditioner for fine-tuning later and this paper says the same thing so basically in in the good old days what you would do is you would devise a method that somehow takes you know takes in a devise a method that takes in a mini batch and in the mini batch you have your data samples and then some of them would be labeled right here you'd have Y and here it'd have a Y but most of them would be not labeled and you'd


have like some sort of loss function that would put special weight on the ones that are labeled or somehow handle these ones that are unlabeled in a way you might be doing like a some sort of a consistency loss such that if they are very nearest near neighbors to these in the feature space they should have similar labels or things like this so these semi-supervised methods they basically try to solve the problem at once but while taking data that is labeled and not labeled this paper goes into a different direction this paper


says first we should it's actually three stages right here and they have a diagram so I don't need to draw they have a three stage approach three stages the one on the left is unsupervised pre training so they say let's forget about the labels right now even like your unlabeled data so even the data where we have the labels let's forget about the labels and let's just do unsupervised pre-training now unsupervised pre training in this kind of setting is also known as self supervised pre training and this first stage is done using a


contrasted loss and that's very similar to sim clear to this contrastive loss so what you'll do and they describe it very very well here so what you'll do is given a randomly sampled mini batch of images each image is augmented twice using random crop color distortion and Gaussian blur creating two views of the same example okay so you have an image in your mini batch each image you take and you make two versions of it and each version you crop real random crop somewhere so version one could be random cropped here version cook two could be


random cropped here and then you put some Gaussian blur on it and so on so a little bit of us as you can see random crop color distortion Gaussian blur so what you want is two different versions of the same image each of these versions has been augmented in a different way cropped in a different way blurred in a different way such it's it's two slightly different versions of the same image and now you want to enforce you want to put this through your network so ultimately as you can see on the right side here what you want


to end up is a a network and then okay we'll forget about this right now what you want to Train is this network right here actually including these projection layers we'll get to them later this is the network that you want to Train so you want to put you take your unlabeled data you take an image you'd make two versions of it and you put those through the network right until the end right here so you'll get Z 1 Z 2 these are the the output of the network for the two images and then what you want to do is you want to take another image that's


not this image and also put it through the network maybe also augment it first and then you have Z 3 so now you have the outputs of 2 things that are supposed to come from the same image and one thing that's supposed to come from a different image and now your loss is simply going to be make those two things close together and push those two things apart or those 3 actually so the loss and this is this is the contrastive loss of self supervised learning as you know you don't need any labels right here you simply say the things that come from the


same image should be close together and the things that come from different images should be far apart and this relies heavily on these data augmentations that you do right here they also employ some other tricks like the momentum encoder from moco from momentum contrast and so on but this is the main the main part so you can pull a lot of strings here to get like another percent of performance but ultimately they won't see the similarity of Zi and ZJ which are the outputs of the same image to be close together and


then this down here they want to be far apart Zi with Z K where K is all the other images okay you can do this in a mini-batch fashion so this is self supervised learning and the reason why you do this is you don't need labels and it tends we know it tends to give very very good representations so um pass that so what this network here will learn will be very good representation with these self supervised loss with contrastive loss for example gives such good performance there have been papers recently that modify the loss and so on


but it's not super well understood yet but if you do it like this there the network here will give you already very very good representation and we know this because we can take a network like this and then simply train a linear classifier on top of that on a data set and achieve very very good performance and mind you you have trained it with unlabeled data right so the the network has never been trained to solve like image net classification it has simply been trained to look at the pictures and determine if you know


two versions of a picture come from the same picture or from different pictures and now if you simply train a linear classifier on top of these representations you're doing extremely well already so we know these representations they actually learn something about these images so that's the first part then stage 2 let's cancel all of that stage 2 is you want to do supervised fine tuning now you already see that the arrow here coming out is not this was a task agnostic big CNN the arrow is actually coming out of those


those yellow boxes and the yellow boxes are these projection heads so in the original seem clear paper what they did was they they wanted originally they wanted to train this network right here this is like a ResNet 50 it's pretty standard in these kind of self supervised approaches and so on to train or these few label approaches to train a standardized network and this is like a ResNet 50 so in the original sim clear paper they said we want to make ResNet 50 as strong as possible but in order to do this loss


right here we are going to attach this projection head just to because the dimensionality here I think is like 2048 and we want to do this inner product in a lower dimension of like maybe 256 or so so this these are just multi-layer perceptrons these are just fully connected layers that compress the representation down to that and once we're done with the unsupervised returning we're going to throw those away right and this ResNet is the thing that we really care about now here they claim okay it actually works better and


they have experiments to prove this or to show this if you use one if you actually leave one of these layers here so in the end they I guess they converge on three projection head layers and then they only throw away the top two and like they make this big deal out of the fact where you know I can just call I can just call this part right here now the encoder and I don't so I don't know exact like I don't see the giant deal here like you've just made your network one layer bigger and now you consider that to be your encoder and the


projection head is now two layers and that will be much easier than calling the projection head three layers but we leave one layer and we train from the middle layers in any case they have this layer additional layer right here compared to the old sim clear and then the representation of that goes into supervised fine-tuning now this is pretty easy this is exactly what it sounds like so now you use only only the dataset that has labels so the part of the data set that has labels and you do the fine tuning and fine tuning is


simply supervised learning you train this network in a supervised fashion on that small fraction of data that has class labels and that already performs pretty well and they show this in experiments but then you can go a step further and do what's known as distillation or self-training and what's distillation or self-training it's so distillation is when you have a network that you call the teacher Network and that network has been trained to do some classification maybe into three classes pretty pretty well okay but now this is


very large and you want maybe a smaller model so you just want like this tiny model because you want to ship it on a mobile device right but it's also supposed to do this and you know that if you just directly train this which is called the student model it doesn't perform as well as the teacher model there is a better way if you have the teacher model you can sort of transfer the knowledge to the student model you can distill the knowledge and how do you do that you do that by so what would you do in supervised training in supervised


training you would take an image put it in and then put the label that comes along with the image you put it up here and you compare the output to the label and that gives you the loss function right now you do that right here if you distill you put the image into both now the teacher is already trained so its output will be a distribution over classes it won't be a single label it will be like okay 90% class 1 10% class 2 0 % class 3 something like this and now you take this as like a pseudo label this entire distribution and you put it


here and you compare the output of the student to that of the teacher and that's your loss function so this kind of the teacher might have learned to put some nuance into the classification well I'm pretty sure this is class one but I'm not a hundred percent sure and it can transfer that knowledge to the student and that makes the student better than had you just trained it from the beginning from from with just the labels right so this is distillation and you can do this even what they call self distillation here or self-training so


apparently this even helps if the teacher is if the student model is the same as the teacher model now why does it help in this case and I think it is not exactly the case in this case because they always say their teacher model has this extra projection layer right and then the student model doesn't have that even if they do self-training but why does it help in this case I mean it's it's kind of shocking and I'm pretty sure it helps in any case but in this particular case it helps because now you're using the unlabeled data


again so you have a teacher model and the teacher model is trained first using unsupervised like this is the teacher model right here using unsupervised training then the teacher model is further fine-tuned on the small data right so it is now already pretty good at the task but how can you get a student model that's even better than the teacher model it's by using again this unlabeled that you have this giant amount of data so what you'll do is you take an image from the unlabeled data and you ask the teacher model teacher


model what do you think about that image right and the teacher model will give you a prediction like let's say again this 90 percent 10% 0% and then you take the student model you input that image and you compare its output to what the teacher said so this combines the teacher model you freeze the teacher model right the teacher model is only trained until here you take it from here the student model is now able to take basically the teacher it takes everything that the teacher model knows not only about this data but about all


the data so it kind of gets to ask the teacher model what do you think about this what do you think about this what do you think about this and it it can incorporate all that knowledge about all of this unlabeled data and that's why the student model here in the end if it's the same size will probably end up even better than the teacher model right so distillation I think also is still kind of a mystery of why you get a better model or I mean to to make it smaller if you make it a lot smaller usually you don't up end up with a


better model but you end up with a pretty good model that you couldn't have gotten by just training the small small model but so that's already pretty cool but why you get a better model with when they're the same size that's I don't think that's well understood yet so that's the three-stage approach so recap first use all of the data without labels to do unsupervised or self supervised contrastive pre-training second use only the data that has labels to do fine tuning third either distill the learnt classifier to a smaller model or distill


it to a model of the same size again within both cases you would again use the unlabeled all of the unlabeled data okay and that's the three-step approach that's same clear v2 in it's in all of its form alright so they go into fine tuning right here and yeah so they say we elaborate with a three layer projection head so that's the three layer projection head this here is the output of resonate 50 where Sigma is a Rayleigh non-linearity and we ignore the bias term for gravity power blah blah blah so they contrast this here for fine


tuning sim clear uses this right here which is just it's basically just a classifier on top of the ow foot of the ResNet 50 okay yada yada yada yada this is fine tuning from the input layer of the projection head to fine tune from the first layer of the projection head we have a new encoder function as this which is ResNet followed by fully connected layers and you see they take the resonate 50 output and they ship it through the first projection layer and then there is a task specific classifier now again why I


don't even see why they make like this ginormous deal out of it especially especially since the last layer of the resonate 50 I'm not ok here is I'm not entirely sure but are they taking the low no they're probably not taking the log it's ok but it's yeah I'm it's just weird like is there even a non-linearity at the end right here or is this really just like to matrix multiplications in a row which I'm going to guess there's a big chance that that's the case that the last layer of this encoder is actually not even followed by non-linearity and


therefore you'll just kind of make the dimension different and I don't see why you can't just just incorporate this into the model and have to like say it over and over again that this is a new special thing right again this is equivalent of tuning from a middle layer of the projection head instead of the output layer like ok you just make your model a bit bigger yeah so the third step is self training or knowledge distillation and they give two variants right here this variant as you can see here this is this is just the


cross-entropy but instead of having labels right here Y or you have the T term what the teacher model thinks Y is given X okay that's that's cross-entropy but not with the true labels but with the output of the teacher model and you can even mix that so you can as you can see right here you can mix this with an actual supervised loss so this would be the supervised loss whatever yeah I guess that I was wrong that wasn't I guess P of Y is always in that case but they don't use this particular kind I think except in one of


the ablations so how does this work it works pretty well and so one of their experiments as you see up here it works pretty well in that if you have 1% of the labels only 1% of imagenet labels which they say is smaller or equal than 13 images per class so there's a thousand classes and you only have 13 labels per class or less if you and they differentiate if your encoder that you train is a resonate 50 then you get and you can see the dashed line here is a supervised baseline you almost get to the supervised baseline with one percent


of the labels and if you actually have a larger ResNet then you get to the supervised performance without without 99 percent of the labels and if you have excuse me ten percent of the labels you pass the supervised baseline so the supervised baseline is on 100% of the labels mind you and you only have ten percent and this outperforms the supervised baseline now of course you could hear you could have another graphic where you show Oh 100% what if we you know what if we do the whole procedure with 100 percent of the labels


so first we don't label the data we do supervised self supervision then we fine-tune on a hundred percent of the data and then we do this distillation again you would of course be even better and I think they have this somewhere in a table but this is already pretty pretty impressive and another claim they make right here is about the model sizes so and this figure is description this now relates to the title they say bigger models yield larger gains when fine-tuning with fewer labeled examples so there there


are three comparative statement words in one sentence let's unpack this bigger models yield larger gains so the bigger the bigger the model the better the good let's say when fine-tuning with fewer labeled examples let's just look at the graph it's pretty it's really clear so here we have a number of parameters going over so these are the different models they look at how many parameters they have to do this whole procedure and here is the relative improvement in percent over the top image net one top accuracy so if you do this whole thing


with a hundred percent of the labels right I'm gonna guess this here this here is where they start out and you can see as you grow your models you grow the performance and this this is just by increasing the model size right you have the same data set you have the same amount of labels you have the same number of steps that you train for and so on just by the fact that you make your model bigger you gain in performance okay now you can see that these curves here are above one another and these curves refer to getting small


less and less labels okay so if you only have 10% of the labels your relative gains are a larger this doesn't mean that you perform better with 10% of the labels than with hundred-percent of the labels that would be that would be like ridiculous well I guess in this day and age nothing is ridiculous but for now we're still performing better by having more labels if we do the same procedure right it's not like here so here this baseline the supervised baseline only does supervised training right so that's why we can outperform it with less of


labels but here we do the same procedure this is relative improvement right so this right here the starting point would be if you had 10 percent of labels and a 25 million model parameter model and this right here for example is if you have the same amount of labels but a 200 million parameter model and this is relative improvement okay but what the graph says is that the relative improvement is larger that the relative improvement is higher the more parameters you have which is the more you go to the right and that effect in


itself is higher the fewer labels you have which is the different graphs and you can see that right here so if you have fewer and fewer labels it becomes more and more important that you have bigger models and that's really counterintuitive right because you would expect that the bigger models they can over fit much more easily to the fewer labels but that doesn't seem the case so this self supervision it really seems to be sort of a counter to this notion of overfitting and if you have larger and larger models that's what they argue in


the paper you might be able to learn more and more features that might be useful for classification so if you have a larger model you might you're gonna learn more kinds of features and then you're going to outperform because you have more chance that these features are going to be useful for classification and I don't think they really make a statement as to why that happens more with the if you have less labels so let's think about this if I have very few labels very very few labels why does it help me even more if I have a big


model well with the same argumentation we could say and maybe they actually say this already so I'm I might be copying them involuntarily maybe with fewer and fewer labels like let's say we have all the labels that's probably too many right if we can learn a task with some accuracy we probably had too many labels okay it's like weakly like we can't learn a task we know we have too few somewhere there is a border where we have enough but that's like can one number and everything else is too too many technically speaking like


learning theoretically speaking so usually we have too many labels and what does that mean that probably means that there are multiple ways like if we have too many labels there are multiple different features we can pick up tool and there are multiple different paths to learn our goals so if we have imagenet and like that there's this weird task to recognize a three and we get lots and lots and lots of examples of threes right we can we can decide on a feature we can say oh I all the threes that I see they have this bow down here


or all the threes that I see they have this bend here and so on but if I only have very few labels there might only be like a single feature that is even theoretically possible to learn from the labels I'm given and therefore if I have a bigger model in cell in pre-training because the pre-training happens with the same amount of data right if I have a if I have a bigger model that does the self supervised preaching is going to learn more features and then there's a higher chance that that one feature that I'm that these very few labels that I am


able to learn something from is going to be in these features so that's kind of how I make sense of it in combination what with what they're saying right here okay so this was the main points they do a lot of empirical studies showing the effects of these sizes they've stressed that it's important to have both deep and wide no that works and they also do this additional attention mechanism over the convolution filters I don't want to go into that particularly but they they also do linear evaluation compared to


supervised compared to to fine tuning on with 100% of the labels so they do a very thorough empirical investigation and yeah I I do appreciate that and they kind of show the same things and here they show the number of layers in the projection head so as you increase the number of layers in the projection head and train from the optimal layer in the middle your performance goes up as you can see but it also this effect is stronger when you have fewer labels right you can see the differences here are greater than the differences here or


even here when you have a hundred percent of the labels so the fewer labels the fewer the labels the more benefit you have from the architecture right here and here they show that it's not always optimal to train from the last projection layer but here the first one so they I guess they converge on three projection layers and you always want to keep the first one around after self supervised training as we mentioned before okay they investigate different different distillation losses and show that it is actually important that you


do the distillation loss on labeled and unlabeled sets you can see here if you only do it if you only train with the labels after fine-tuning you get poor performance if you do the label and distillation loss but only do it on the data set where you have labels then you get more performance if you do label and distillation loss but also include your unlabeled data you get even more performance and then if you do that but you don't do the label loss so before we've seen you can mix the distillation loss with the label loss if you have


lots of labels then you drop in performance again and you can see right here the drop in performance is proportional to how many labeled examples you haven't that's that's natural right if you have the labels you can actually mix that information in with the distillation loss and that make you better and here they drop point one percent and here they drop less than one percent by leaving away the labeled but their point basically is that it is more important to to distill using also unlabeled data than it is to


distill including the label loss and it's much easier to not include the label or so they don't do it I guess alright so I think that was it they they compared as I said they compare like self distillation where you distill into an equally sized model and down distillation where you distill into a smaller model maybe that's vice-versa and they do a lot of comparison to other methods so this is a very thorough work I feel and yeah if you want more about the exact experiments I invite you to look at the paper and let's just have a final look


at the broader impact statement right here so the broader remember the broader impact statement is supposed to to force you to think about how society might be impacted at large by your work so it says the finding described in this paper can potentially be harnessed to improve accuracy in any application or computer vision where it is more expensive or difficult to label additional data than to train larger models such applications are clearly beneficial to society for example in medical applications we're acquiring high quality labels requires


careful annotation by clinicians better semi-supervised learning approaches can potentially help save lives application of computer vision to agriculture can increase crop yields which may help to improve availability of food however we also recognize their approach could become a potential component of harmful surveillance systems more over there is an entire industry built around human labeling services and technology that reduces the need for these services could lead to short-term loss of income for some of those currently employed or


contracted to provide labels so ask yourself how much of that statement has to do with the actual novelty this paper and the answer is of course zero right like you can replace like our method in this thing with like machine learning or computer vision in general like oh really sim clear v2 specifically can increase crop yields like that specific invention of this paper will lead to higher crop yields will lead to surveillance systems so I'm yeah you know I I think like I'm not gonna get too too upset about least I mean this I


think it's quite funny but just again I I've I wonder whether the people advocating for these things are happy with these statements because clearly clearly this is just a template that you copy/paste from paper to paper replacing like a few words and if it's computer vision you're like oh my deep fakes and if it's NLP it's like ah fake news and yeah I I wonder if really anything like particularly is has I wonder whether these people are happy now yeah I just I wonder and if if they are I wonder whether it's really for the reason that


today I want to talk to you guys about how to get machine learning on to your physical devices when there is no network connectivity and convince you why you would want to do that why you would actually prefer even with network connectivity to leave it aside perhaps in certain situations but to get there I want to start back we're gonna go back in time back to when the computer first came about right that started this computing revolution that we're in today but today we're in what we call an AI first world how did we get here well

On-device Machine Learning With TensorFlow


first we had the computer and that gave us computing then the internet came right the internet connected all the computers together and I want to point out that the internet did not wipe out the computer right the computer is still used today then came mobile the mobile revolution can brought the Internet to everyone it brought the connected computers into our pockets and again we see that mobile did not wipe out the Internet we still use the Internet it's still strong with an IP version 6 we're running out of IP


addresses and so too will a I not replace mobile but build on top of it and so I see mobile has a key foundation for the core of how we will interact with AI in the years to come and so to that end I want to show you guys one approach for integrating AI into mobile devices and bring amazing user experiences today over half of the fortune 500 globally have disappeared the company's kaput gone since 2000 and so how can we not have that situation first of all but also what will happen to the other half right it's the


companies who embrace AI these startups who are well nowadays all the startups are saying we are AI startup everyone is a start-up but a few short years ago machine learning and AI was something that companies would add on to their product they would say oh yeah we do a little machine learning here on the side right but now I root everyone's doing it it's it's super popular and for the most part people trained on the server right server has a lot of compute power it makes sense and I'm now going to refute that because it


makes sense Mobile is not a compute powerful platform so we train on the server that's fine but what if we did the predictions on the mobile on mobile what if we did inference there instead of serving them from a server web server and in particular this approach can lead to amazing user experiences I want to just show with one illustration example this is a Google Translate and it has the ability to overlay the translation directly in the image that you see now you can tell just from the the speed that it can do this that it is


definitely not happening over the network right the video is not being streamed to Google servers and then sent back that's why it works when you are offline that works it works on a boat it works underwater well if your phone can be underwater it would even probably would work in space and having access to AI on your phone wherever you go that is responsive and accurate that can really change things on a global scale for mobile for the internet and for computing so how can we make something like that right machine learning is hard


enough by itself then put it on mobile I mean mobile apps aren't easy either and so combining those that can be a real challenge so let me start by showing you guys what a little demo of what I've kind of put together and then we'll talk through how we might build something like this it's a simple demo just mainly just to demonstrate the core kind of functionality and what I've got here is a phone and I have a little app if we switch over to the camera here I have on the table a few different candies let's see what we see here oh great and so


let's see you guys see this okay so what I'm gonna do is what we'll take some of these away and if the lighting is good you know here we have a a Reese's Cup right and we can see here know that as the image kind of isolates down it recognizes that no I also have a smaller Reese's Cup which just fell on the floor okay so we have a smaller Reese's Cup here and you know and we it will switch over and recognize that now you might say well how do I know that he's not cheating by sending all these images over the web right well let me first of


all there's no Wi-Fi here secondly let's let's just go ahead and hit airplane mode right as well and we'll see here that you know everything continues to work just fine these are some more peanut butter cups I'm a big fan of peanut butter and so let's push these guys away and so here we have a shot of the Justin's see here it says white chocolate peanut butter cups and you know one of my hand enters the frame it may get upset candy is a little bit bent out of shape from its days inside the suitcase but you know you can see that


it clearly recognizes that versus a very similar packaging right but this is milk chocolate this is a milk chocolate peanut butter cup and and this one also updates you know it updates right away you can see the the confidence and then we here we have some juicy fruit gum just for variety you know some people don't like peanut butter I understand that so that's that's the gist of the demo here and so if we switch back to the slides we can think about how we might build something like this how do we go from collecting data to having an app that


can recognize images in real time custom images right I just I chose these pretty arbitrarily if you were to take any generic machine learning visual model and pointed them at this it might take candy candy bar maybe it even says chocolate maybe it just says yellow but how can you get something to recognize something that's specific you know imagine having something recognize your particular products your whether it's your brand things in your home and so this is my are kind of guidelines here this is what will follow this our


little nap this is our road map and what we're gonna do is we're gonna try to fill in each of these blocks and go from gathering data to having an application so the first thing you got to do is collect data my data collection is as we all know the most fun step of machine learning now I've found a bit of a shortcut for you instead of you know collecting lots of pictures and then trimming them down and then labeling them and it would just be a lot of work right it's a lot of pictures so maybe we can shoot some video we shoot some video


and we only take the video of that particular object so I go through each one now you can see there I have one of the the peanut butter cups and we go through each one and we capture a video and what's nice about that is that entire video every single frame is a picture of that object hopefully from a different angle so keep that camera moving and then what we can do is we can well we chop that up right there's a command line tool called ffmpeg and there's lots of tools ways you can chop up a video that's a solved problem and


we put those pictures all into one place all together in one folder for each of the objects you want to recognize so so one folder for your juicy fruit one folder for your milk chocolate peanut butter cups one folder white chocolate ones and so on and so now we've effectively just labeled all of the images right we didn't have to come up with any sophisticated way to label it for us with some system so that's great so we have folders of images what's next well we take these pictures and we send them to training


right so in my particular case I uploaded them to the cloud because my MacBook was running out of space form all the images and pictures and so I zipped them up put them in the cloud and I happen to do my training in the cloud you can do them on your data center you can do them on your local machine if you have a lot of storage and the training we did is using transferred learning now this speaker before me mentioned transfer learning so I won't go too much into detail about it but I do have a kind of little story a little analogy


that I recently accidentally discovered I was playing with a puzzle this is a jigsaw puzzle right lots of pieces and and this is the final picture this is the box showing what I was supposed to build and I'm not a very good jigsaw puzzle person I kind of struggle with it but as I was struggling through this I realized you know there's some good tricks in here I could do first of all I could separate the the pieces with no images the white pieces from the everything else right so I moved all of the white pieces to one side okay so


that's the obvious step first step but then what how do we start from there I also noticed that in this image the roofs all the individual roofs are very distinctly patterned and so I said ah I know I can look for pieces with this pattern and with that I can begin to put those together those were easy to find I could recognize those patterns so my brains neural network through my eyes could find those individual small pieces and combine them together and say ah there's a roof similarly I noticed in the stairs the stairs have a distinct


pattern they're parallel lines there's some of those X's there and I could find the pieces that kind of looked similar and begin to squeeze those together and eventually get that together and instantly everything else came together well there was a few more steps right but that that's as far as I got the edge pieces they're so hard so so a convolutional neural network kind of works in a similar way I use transfer learning with the inception model which is a model that the Google brain team created a few years ago it has 48 layers


and to give you some perspective on how big or how small that might be just a few years ago I think it was 2011 or 2012 it was Impractical to train a neural network that was more than four layers deep the year before inception v3 came out the model that won the International image recognition competition called imagenet that had 22 layers so in the inception of e3 model really was a giant leap ahead right more than double the number of layers it really showcased the improvements in both computational power that was


available as well as network design so we can use that wonderful research and take it for our advantage so we trained the last layer right of the network and leave everything else intact this means that everything in the visual part of recognizing those pieces recognizing the little bits the principal pieces are already there in place for us so you have a great model you can train it but when you're done you look at your file system and you say wow this model is 84 megabytes I'm trying to put this in a mobile app


can you help me out sure thing we're gonna optimize it for mobile what can we do to shrink down a model well handily enough there is a graph transform tool and what that's gonna do for us there's a couple of steps in there that we can do the first thing is a technique called quantizing or quantization and the floating-point numbers those 32-bit floating-point numbers that are taking up all this space we're gonna shrink that down to just eight bits how can we get away with this we're gonna lose so much accuracy right


well not necessarily luckily neural networks are designed for fuzziness in their inputs so by quantizing down to eight bits and and that's just by rounding but like actually saying these numbers are close enough we're gonna make them all kind of say this number and then this number so we then say the range we're gonna split that out but into 256 pieces so if the range of values is say negative ten to thirty within there when they divided up into 256 little steps and so that gives you a little more accuracy than just purely


changing it directly to 8-bit and additionally that means when you do compression those values are the same and so that's what it lets you go down literally for X so you go from 84 megabytes down to around 20 21 megabytes and one small additional thing you can do is take away the parts of the graph the parts of the graph that you don't need anymore for prediction there's some graph pete nodes which are only useful for training and there's also a tool that will prune that down for you as well so that's also part of this graph


transform tool is a whole suite of tools so that's really useful and I also want to call out that so far everything we've done is basically running existing code and existing tools you didn't have to custom write anything the only custom thing you had to do was shoot that video and run ffmpeg so this really makes it you know puts it in an immediate possibility type of stage there is one more thing one more consideration to think about when looking at deploying a machine learning model to a mobile device and that is whether you package


it inside the app or alongside the app you can make it a data file or you can make it integrate it into the app and some of the thoughts there are whether you want to be able to secure the model whether you want to be able to download updates without pushing a new updated version of the app itself and whether or not you care about sizing and how whether or not you want to secure the the model from outside access so that's our our overall design right we gather it up we shoot up the videos slice it up and and train and optimize


and then we can deploy so that that's our of our finished model and this is a video of the the same thing so I won't show that and the final kind of point here is how are we going to be able to how have we done this right what makes it possible and that's tensor flow tensor flow is Google's machine learning library hopefully some of you have heard of it it's too dark so I can't do a show of hands but it's been incredible to see the community adoption and the reception to the launch it was open sourced in November of 2015 and hit 1.0 this past


February and with that we have support for not just these platforms right which we expect CPUs GPUs and of course Android but also iOS and Raspberry Pi so for those of you who like to tinker with IOT devices you can load a model onto a Raspberry Pi so that means you can recognize things without any network traffic it can be handy and the community responds to tensorflow in the past while now more than fourteen months but in the first fourteen months there were over fourteen thousand commits hundreds of non-google


contributors and now that it's 1.1 of 1.0 it is production ready the AP's api's are stable and backwards compatible so things won't change out from under you so that's really quite nice and so in conclusion putting machine learning on mobile will just make that experience of mobile the internet and computing that much more powerful and usher in a new wave of innovation and open a whole new world of possibilities and moreover you can build one easily by gathering your own labelled data simply by shooting a video


Probably a lot of you know the story of the two salesmen who went down to Africa in the 1900s. They were sent down to find if there was any opportunity for selling shoes, and they wrote telegrams back to Manchester. And one of them wrote, "Situation hopeless. Stop. They don't wear shoes." And the other one wrote, "Glorious opportunity. They don't have any shoes yet." (Laughter) Now, there's a similar situation in the classical music world, because there are some people who think that classical music is dying.

The transformative power of classical music | Benjamin Zander


And there are some of us who think you ain't seen nothing yet. And rather than go into statistics and trends, and tell you about all the orchestras that are closing, and the record companies that are folding, I thought we should do an experiment tonight. Actually, it's not really an experiment, because I know the outcome. (Laughter) But it's like an experiment. Now, before we start -- (Laughter) Before we start, I need to do two things. One is I want to remind you of what a seven-year-old child sounds like when he plays the piano.


Maybe you have this child at home. He sounds something like this. (Music) (Music ends) I see some of you recognize this child. Now, if he practices for a year and takes lessons, he's now eight and he sounds like this. (Music) (Music ends) He practices for another year and takes lessons -- he's nine. (Music) (Music ends) Then he practices for another year and takes lessons -- now he's 10. (Music) (Music ends) At that point, they usually give up. (Laughter) (Applause) Now, if you'd waited for one more year, you would have heard this.


(Music) (Music ends) Now, what happened was not maybe what you thought, which is, he suddenly became passionate, engaged, involved, got a new teacher, he hit puberty, or whatever it is. What actually happened was the impulses were reduced. You see, the first time, he was playing with an impulse on every note. (Music) And the second, with an impulse every other note. (Music) You can see it by looking at my head. (Laughter) The nine-year-old put an impulse on every four notes. (Music) The 10-year-old, on every eight notes.


(Music) And the 11-year-old, one impulse on the whole phrase. (Music) I don't know how we got into this position. (Laughter) I didn't say, "I'm going to move my shoulder over, move my body." No, the music pushed me over, which is why I call it one-buttock playing. (Music) It can be the other buttock. (Music) You know, a gentleman was once watching a presentation I was doing, when I was working with a young pianist. He was the president of a corporation in Ohio. I was working with this young pianist, and said,


"The trouble with you is you're a two-buttock player. You should be a one-buttock player." I moved his body while he was playing. And suddenly, the music took off. It took flight. The audience gasped when they heard the difference. Then I got a letter from this gentleman. He said, "I was so moved. I went back and I transformed my entire company into a one-buttock company." (Laughter) Now, the other thing I wanted to do is to tell you about you. There are 1,600 people, I believe. My estimation is that probably 45 of you


are absolutely passionate about classical music. You adore classical music. Your FM is always on that classical dial. You have CDs in your car, and you go to the symphony, your children are playing instruments. You can't imagine your life without classical music. That's the first group, quite small. Then there's another bigger group. The people who don't mind classical music. (Laughter) You know, you've come home from a long day, and you take a glass of wine, and you put your feet up. A little Vivaldi in the background doesn't do any harm.


That's the second group. Now comes the third group: people who never listen to classical music. It's just simply not part of your life. You might hear it like second-hand smoke at the airport ... (Laughter) -- and maybe a little bit of a march from "Aida" when you come into the hall. But otherwise, you never hear it. That's probably the largest group. And then there's a very small group. These are the people who think they're tone-deaf. Amazing number of people think they're tone-deaf. Actually, I hear a lot, "My husband is tone-deaf."


(Laughter) Actually, you cannot be tone-deaf. Nobody is tone-deaf. If you were tone-deaf, you couldn't change the gears on your car, in a stick shift car. You couldn't tell the difference between somebody from Texas and somebody from Rome. And the telephone. The telephone. If your mother calls on the miserable telephone, she calls and says, "Hello," you not only know who it is, you know what mood she's in. You have a fantastic ear. Everybody has a fantastic ear. So nobody is tone-deaf. But I tell you what. It doesn't work for me to go on with this thing,


with such a wide gulf between those who understand, love and are passionate about classical music, and those who have no relationship to it at all. The tone-deaf people, they're no longer here. But even between those three categories, it's too wide a gulf. So I'm not going to go on until every single person in this room, downstairs and in Aspen, and everybody else looking, will come to love and understand classical music. So that's what we're going to do. Now, you notice that there is not the slightest doubt in my mind


that this is going to work, if you look at my face, right? It's one of the characteristics of a leader that he not doubt for one moment the capacity of the people he's leading to realize whatever he's dreaming. Imagine if Martin Luther King had said, "I have a dream. Of course, I'm not sure they'll be up to it." (Laughter) All right. So I'm going to take a piece of Chopin. This is a beautiful prelude by Chopin. Some of you will know it. (Music) Do you know what I think probably happened here? When I started, you thought, "How beautiful that sounds."


(Music) "I don't think we should go to the same place for our summer holidays next year." (Laughter) It's funny, isn't it? It's funny how those thoughts kind of waft into your head. And of course -- (Applause) Of course, if the piece is long and you've had a long day, you might actually drift off. Then your companion will dig you in the ribs and say, "Wake up! It's culture!" And then you feel even worse. (Laughter) But has it ever occurred to you that the reason you feel sleepy in classical music is not because of you, but because of us?


Did anybody think while I was playing, "Why is he using so many impulses?" If I'd done this with my head you certainly would have thought it. (Music) (Music ends) And for the rest of your life, every time you hear classical music, you'll always be able to know if you hear those impulses. So let's see what's really going on here. We have a B. This is a B. The next note is a C. And the job of the C is to make the B sad. And it does, doesn't it? (Laughter) Composers know that. If they want sad music, they just play those two notes.


(Music) But basically, it's just a B, with four sads. (Laughter) Now, it goes down to A. Now to G. And then to F. So we have B, A, G, F. And if we have B, A, G, F, what do we expect next? (Music) That might have been a fluke. Let's try it again. (Music) Oh, the TED choir. (Laughter) And you notice nobody is tone-deaf, right? Nobody is. You know, every village in Bangladesh and every hamlet in China -- everybody knows: da, da, da, da -- da. Everybody knows, who's expecting that E. Chopin didn't want to reach the E there,


because what will have happened? It will be over, like Hamlet. Do you remember? Act One, scene three, he finds out his uncle killed his father. He keeps on going up to his uncle and almost killing him. And then he backs away, he goes up to him again, almost kills him. The critics sitting in the back row there, they have to have an opinion, so they say, "Hamlet is a procrastinator." Or they say, "Hamlet has an Oedipus complex." No, otherwise the play would be over, stupid. (Laughter) That's why Shakespeare puts all that stuff in Hamlet --


Ophelia going mad, the play within the play, and Yorick's skull, and the gravediggers. That's in order to delay -- until Act Five, he can kill him. It's the same with the Chopin. He's just about to reach the E, and he says, "Oops, better go back up and do it again." So he does it again. Now, he gets excited. (Music) That's excitement, don't worry about it. Now, he gets to F-sharp, and finally he goes down to E, but it's the wrong chord -- because the chord he's looking for is this one, and instead he does ...


Now, we call that a deceptive cadence, because it deceives us. I tell my students, "If you have a deceptive cadence, raise your eyebrows, and everybody will know." (Laughter) (Applause) Right. He gets to E, but it's the wrong chord. Now, he tries E again. That chord doesn't work. Now, he tries the E again. That chord doesn't work. Now, he tries E again, and that doesn't work. And then finally ... There was a gentleman in the front row who went, "Mmm." (Laughter) It's the same gesture he makes when he comes home